(19) 




Europdisches Patentamt 
European Patent Office 
Office europ6en des brevets 






(11) 



EP 0 893 308 A2 



(12) 



(43) Date of publication: 

27.01.1999 Bulletin 1999/04 

(21) Application number: 98113679.9 

(22) Date of filing: 22.07.1998 



EUROPEAN PATENT APPLICATION 

(51) IntCI. 6 : B60R 16/02 



(84) Designated Contracting States: 

AT BE CH CY DE DK ES Ft FR GB GR IE IT LI LU 
MC NL PT SE 

Designated Extension States: 
AL LT LV MK RO SI 

(30) Priority: 22.07.1997 J P 212491/97 

27.02.1998 JP 64414/98 
12.03.1998 JP 82626/98 
1 3.03.1998 JP 82710/98 
23.03.1998 JP 95386/98 
23.03.1998 JP 95387/98 

(71) Applicant: 

Kabushiki Kaisha Equos Research 
Tokyo 101 (JP) 



(72) Inventors: 

• Kubota, Tomoki 
Arakawa-ku, Tokyo-to (JP) 

• Hon, Kojf 
Bunkyo-ku, Tokyo-to (JP) 

. • Mazda, Manabu 

Ichikawa-shi, Chiba-ken (JP) 

• Adachi, Kazuhide 
Nerima-ku, Tokyo-to (JP) 

• Mukai, Kouji 
Suginami-ku, Tokyo-to (JP) 

(74) Representative: 

VOSSIUS & PARTNER 
Siebertstrasse 4 
81675 MOnchen (DE) 



CM 
< 

00 



CO 
CO 

o> 

CO 



(54) Device mounted in vehicle 

(57) A device is mounted in a vehicle which allows 
one or more of personified agent or imaginary living 
body to appear before a driver or passengers in the 
vehicle for communication therewith. A current status 
sensor (40) detects predetermined vehicle current sta- 
tus data, and a memory (292, 293) stores study data in 
response to entry of the latest current status data. One 
of a plurality of communication programs stored in a 
program table; (29 1 ) may be selected in accordance with 
the current status data and/or the study data. An agent 
control unit (1j1); Controls activity of the agent in accord- 
ance with the fleeted communication program. The 
determined agerit activity is outputted through a display 
(27) and a speaker (25) for communication with the 
driver. 
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Description 

This invention relates to a device mounted in a vehi- 
cle, and more particularly to a device for controlling 
activity of a personified agent on a display in a vehicle 
room for communication with a user such as a driver. 

Various audio instruments such as radio, cassette 
tape player, CD player, MD player, etc. are installed in a 
vehicle to provide comfort and relaxation to the driver 
while driving the vehicle. This is only one-way communi- 
cation. 

Bi-directional communication between the driver 
and his or her friends outside of the vehicle may be 
achieved by using radio communication medium such 
as radio transceiver and portable telephone. However, 
the driver have to find out a companion to talk to by 
operation of the radio communication medium, which 
might be dangerous during driving. The companion out- 
side the vehicle can not understand the vehicle and 
driving condition that the driver meets with. 

In Japanese patent laid-open publication No. 9- 
102098, specific information is transmitted to the driver 
through corresponding expression and/or activity of an 
agent appearing on a display. In accordance with this 
prior art, the same expression and/or activity of the 
agent is appearing in the same situation. Thus, this pro- 
vides improved visual literacy but is still a kind of con- 
ventional meters. 

Accordingly, it is an object of the present invention 
to provide a novel device mounted in a vehicle that pro- 
vides bi-directional communication with the user in the 
vehicle. 

Another object of the present invention is to provide 
bi-directional communication with the user in a vehicle 
by means of a personified agent whose activities are 
determined not only by the current vehicle condition but 
also by the results of past record study. 

To achieve the above object, in accordance with the 
present invention, a device is mounted in a vehicle 
which allows one or more of personified agent or imagi- 
nary living body to appear before a driver or passengers 
in the vehicle for communication therewith. 

In accordance vyith an aspect of the present inven- 
tion, the devicej comprising first sensor means for 
detecting predetermined current status; first memory 
means for storing study data renewable in response to 
entry of the latest current status detected by the first 
sensor means; at least one personified agent communi- 
catable with user in the vehicle; second memory means 
for storing a plurality of communication programs; com- 
munication program select means for determining one 
of the communication programs to be executed in 
accordance with the current status detected by the first 
sensor means and/or the study data currently stored in 
the first memory means; agent control means for con- 
trolling activity of the agent in accordance with the 
selected communication program; agent output means 
for outputting the agent's activity to the user in the vehi- 



cle determined by the agent control means. 

In a preferred embodiment, the first sensor means 
detects at least one vehicle condition selected from the 
group consisting of a current vehicle position, a current 

5 vehicle speed, a current shift position, winker operation, 
wiper operation, a coolant temperature, an engine oil 
temperature, an engine oil pressure, a passenger room 
temperature, an open air temperature, ignition switch 
operation, a brake pedal stroke, an accelerator pedal 

io stroke, side brake operation, horn operation, belt loos- 
ening, a fuel remainder, light or lamp operation, seat 
belt operation, door opening, window opening, ABS 
operation, air-conditioner operation, distance to the pre- 
ceding and succeeding vehicles and decrease of air 

is pressure in tire. 

The user's own data may be included in the current 
status. The user's own data may also be considered in 
determining the communication program to be exe- 
cuted. 

20 The device of the present invention may further 
comprises second sensor means for detecting user's 
response to the agent's activity outputted by the agent 
output means, and third memory means for storing 
user's response data detected by the second sensor 

25 means. In this embodiment, the user's response data is 
also considered in determining the communication pro- 
gram to be executed. The user's response data is sub- 
jected to study so that the study data in the first memory 
means is also renewable in response to entry of the lat- 

30 est user's response data. 

The agent output means may comprise one or both 
of display means and speaker means, fn a preferred 
embodiment, utterance pattern of the agent outputted 
by the speaker means is selectively determined by the 

35 agent control means. In another preferred embodiment, 
appearance pattern of the agent outputted by the dis- 
play means is selectively determined by the agent con- 
trol means. 

The device may include agent's character select 

40 means for determining a specific character to be given 
to the agent. In this embodiment, the agent control 
means determines the agent s activity in consideration 
of the agent's character determined by the agent's char- 
acter select means. 

45 The device may include fourth memory means for 
storing data indicating user's familiarity to agent opera- 
tion of the device. The agent control means determines 
the agent's activity in consideration of the agent's famil- 
iarity stored in the fourth memory means. 

so In another preferred embodiment, the agent output 
means allows a plurality of different agents to be distin- 
guishably recognized one another by the user in the 
vehicle. For example, the device includes fifth memory 
means for storing sub-agent program, which is selected 

55 in correspondence to a specific communication pro- 
gram stored in the second memory means so that a 
sub-agent of the selected sub-agent program is output- 
ted by the agent output means together with the main 
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agent of the executed communication program. The 
sub-agent may be a help-agent explaining the activity of 
the main agent. The sub-agent may be a short-cut 
agent allowing short-cut of operation executed by the 
main agent communication program. Preferably, the 5 
sub-agent program becomes operative when data indi- 
cating user's familiarity to agent operation of the device 
is greater than a predetermined critical value. 

In another preferred embodiment, a background of 
the display means is selectively determined so that the 10 
agent appears on the display means together with the 
selected background. 

The above and other objects of the present inven- 
tion will become apparent from the following description 
when read in conjunction with the accompanying draw- 75 
ings in which: 

Fig. 1 is a block diagram showing an agent system 
in accordance with a first embodiment of the 
present invention ; Fig. 2 is an example of a program 20 
select table; Fig. 3 is a table describing agent's 
activities to be outputted when a specific program 
number is selected in reference to Fig. 2; Figs. 4A- 
4D represent a series of an agent's activity shown 
on a display by way of example; Fig. 5 is an exam- 25 
pie of a study item data in an agent data memory of 
' Fig. 1; Figs. 6A and 6B represent an example of 
response data in the agent data memory; Fig. 7 
shows various data files stored in a navigation data 
memory; Fig. 8 shows various sensors in a current zo 
status sensor; Fig. 9 is a flowchart of the agent con- 
trol operation executed in this embodiment; Figs. 
1 0A-10C show an example of the agent activity out- 
putted in this embodiment; Figs. 11A-11C show 
another example of the agent activity outputted in 35 
this embodiment; Fig. 12 shows contents of the 
agent data memory in a second embodiment; Fig. 
13 is an example of an agent character select table 
in the second embodiment; Fig. 1 4 is an example of 
a voice output pattern select table in the second 40 
embodiment; Fig. 15 is a flowchart showing a main 
routine of th4 agent control operation in the second 

1 

embodiment^; Fig. 16 is a flowchart showing a sub- 
routine ; executed at step S114 in Fig. 15, Fig. 17 
shows 'f contents of the agent data memory in 45 
another modified embodiment; Fig. 18 is a flow- 
chart showing the agent control operation to be car- 
ried out at S114 in Fig. 15; Figs. 19A-19C show an 
example of the agent activities outputted in this 
embodiment; Fig. 20 A represents a example of a so 
program select table; Fig. 20B represents the 
agent's activities to be outputted in the communica- 
tion program selected by the table of Fig. 20A; Fig. 
21 shows contents of the agent data memory in still 
another modified embodiment; Fig. 22 shows an ss 
example of the familiarity data in the agent data of 
Fig. 21; Fig. 23 shows contents of the agent data 
.memory in still another modified embodiment; Fig. 



24 is an example of the help-agent select table in 
the agent data memory of Fig. 23; Fig. 25 shows 
the help-agent activities to be outputted in the cor- 
responding help-agent program number in the table 
of Fig. 24; Fig. 26 is a flowchart of the main routine 
of the agent control operation in accordance with 
this embodiment; Figs. 27A-27C illustrate an exam- 
ple of the agent control operation made by the flow- 
chart of Fig. 26; Fig. 28 shows contents of the agent 
data memory in still another modified embodiment; 
Fig. 29 is an example of the sub-agent program 
table in the agent data memory of Fig. 28; Fig. 30 
shows short-cut agent activities to be outputted in 
the corresponding short-cut agent program number 
in the table of Fig. 29; Fig. 31 is an example of the 
contents of the frequent program data in the agent 
data memory of Fig. 28; Fig. 32 is a flowchart of the 
main routine of the agent control operation in 
accordance with this embodiment; Fig. 33 is an 
explanatory view showing multi-stage operation of 
the communication program for menu selection; 
Fig. 34 is an explanatory view showing the short-cut 
agent activity incorporated into the multi-stage 
operation of Fig. 33; Fig. 35 is a flowchart of the 
short-cut agent operation that is a sub-routine 
branched from the main routine of Fig. 15; Fig. 36 
shows an example of the display when the short-cut 
agent appears thereon together with the main 
agent; Fig. 37 shows an example of the display 
when only the main agent appears thereon; Fig. 38 
shows contents of the agent data memory in still 
another modified embodiment; Fig. 39 shows an 
example of the study item data in the agent data 
memory of Fig. 38; Fig. 40 shows an example of the 
background select table in the agent data memory 
of Fig. 38; Figs. 41A-41D show examples of the 
selected background that is shown on the display 
together with the prescribed agent's activity; Fig. 42 
shows an example of the response recognition data 
in the agent data memory of Fig. 38; Fig. 43 is a 
flowchart of the agent control operation to be exe- 
cuted in this embodiment as a sub-routine 
branched off from the main routine of Fig. 15; Figs. 
44B and 44C show pictures successively appearing 
on the display on the conditions of Fig. 44A in this 
embodiment; Fig. 45 shows contents of the agent 
data memory in still another modified embodiment; 
Fig. 46 shows an example of the appearance select 
table in the agent data memory in Fig. 45; Fig. 47 is 
a flowchart of the agent control operation to be exe- 
cuted in this embodiment as a sub-routine 
branched off from the main routine of Fig. 15; Figs. 
48A-48C show an example of the agent's appear- 
ance and activity appearing on the display in this 
embodiment; Figs. 49A-49B show another example 
of the agent's appearance and activity in this 
embodiment; and Fig. 50 show still another exam- 
ple of the agent's appearance and activity in a mod- 
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ified embodiment. 

The first embodiment of the present invention will 
be hereinbelow described in more detail in reference to 
Figs. 1-11 of the accompanying drawings. s 

In this embodiment, a personified agent appears on 
a graphic display in a vehicle. The agent may be repre- 
sented as a plane image or three-dimensional image by 
holography, for example. The activity (including utter- 
ance) of the agent is controlled in response to the vehi- 10 
cle condition and the result of past record study. The 
vehicle condition may include various condition of the 
vehicle itself, the driver, thepassenger(s) and preceding 
and succeeding vehicles. The past record study means 
study of past record of the vehicle condition and may is 
include reaction of the user (driver) to the agent's activ- 
ity appearing on the display. Thus, the driver may com- 
municates with the agent in the vehicle, even while 
driving. 

The "agent" used herein is defined as an imaginary 20 
personified subject who can make judgement and study 
like a human body. Accordingly, the same agent makes 
different communication with the driver on the different 
vehicle condition. Even on the same vehicle condition, 
the past record study may result in different communica- 25 
tion by the agent. The agent is not a superhuman and 
may sometime make mis-judgement and blunder as fas 
as it does not affect the vehicle driving. Such mis-judge- 
ment and blunder made by the agent may be corrected 
in accordance with the driver's reaction, which is a kind 30 
of the past record study. 

The agent represented on a display may be like a 
human body. It may also be like an animal such as a 
chick, dog, cat, hog, rat, etc. It may also be a robot or 
any character. The agent may be a child at the begin- 35 
ning and is grown with time. The appearance and voice 
of the agent may be selected from predetermined plural 
ones. 

Fig. 1 is a block diagram of the device embodying 
the present invention. There is a processing unit 1 40 
including a navigation processing unit 10 for finding out 
a drive route' to a predetermined goal that is known to 
the driver by visual} and/or voice message, an agent 
processing unit fHi-for controlling the activity of the 
agent, an interface .fl/F) 12, an image processing unit 45 
13, a voice control* unit 14 and a current status data 
processing section 15. 

Navigation processing unit 10 and agent process- 
ing unit 11 have respectively a central processing unit 
(CPU) that operates in response to the input information so 
to perform various data processing operation and con- 
trol of parts. To CPU are connected ROM, RAM and 
timer through data bus lines. Navigation processing unit 
1 0 and agent processing unit 1 1 are connected with 
each other by a network so that data processed in one ss 
unit may be given to the other unit. ROM is a read-only- 
memory storing various data and programs necessary 
to perform control in CPU. RAM is a random-access- 



memory to be used as a working memory when CPU 
performs data processing operation. 

CPU of navigation processing unit 10 and agent 
processing unit 1 1 reads out the programs in ROM to 
conduct necessary operation. In modified embodiment, 
CPU reads out the programs from a memory medium in 
a memory drive 23, in which case the programs are then 
installed in an agent data memory 29, a navigation data 
memory 30, a hard disk (not shown) or any other mem- 
ory. A necessary program is loaded from the memory to 
RAM for execution. It may be possible that CPU reads 
out the necessary program from the memory drive 23 to 
load it directly to RAM. 

To navigation processing unit 10 are connected a 
current position sensor 21 and navigation data memory 
30. To agent processing unit 1 1 is connected agent data 
memory 29. An input device 22, memory drive 23 and a 
communication control device 24 are connected to l/F 
12. A display 27 and a camera 28 are connected to 
image processing unit 13. A voice processing unit 25 
and a microphone 26 are connected to voice control unit 
14. A current status sensors 40 are connected to cur- 
rent status data processing section 15. 

Current position sensor 21 for detecting the current 
absolute position (its latitude and longitude) of the vehi- 
cle has a GPS (global positioning system) receiver 211, 
a direction sensor 212, a steering sensor 213, a dis- 
tance sensor 214 and a beacon sensor 215 that 
receives the position signal from a beacon located on 
the road. GPS receiver 211 receives radio waves from 
earth satellites to determine the vehicle position. Bea- 
con sensor 215 receives positional information from 
beacons arranged on roads. GPS receiver 211 and bea- 
con sensor 215 can solely act to measure the vehicle 
position. Where GPS receiver 21 1 can not receive the 
radio waves and beacon sensor 215 can not receive the 
positional information, direction sensor 212 and dis- 
tance sensor 214 cooperates with each other to infer 
the current position. 

Direction sensor 212 includes a terrestrial magnet- 
ism sensor that detects terrestrial magnetism to deter- 
mine orientation of the vehicle, a gyromagnetic sensor 
that detects a rotational angle velocity of the vehicle that 
is integrated to determine orientation of the vehicle, a 
pair of wheel sensors that detect an output pulse differ- 
ence (a difference of movement between left and right 
wheels) to detect an angle of gyration of the vehicle. 
Steering sensor 213 employs an optical sensor or rota- 
tion resistance volume mounted to a rotating member of 
a steering to detect a steering angle. Distance sensor 
214 detects a distance of movement by detecting revo- 
lution of the wheels or acceleration, for example. 

Input device 22 is used to input any data, for exam- 
ple, the current position and the goal when starting nav- 
igation processing. Input device 22 also act as means 
for inputting the driver's response to the agent's mes- 
sage or question. An example of input device 22 is a 
touch panel arranged on display 27 and adapted to 
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input any information by a touch on a key or menu rep- 
resented thereon. Another example of Input device 22 is 
a keyboard, mouse, bar code reader, write pen, joy 
stick, infrared remote controllable input device and voice 
analyzer. An infrared remote controllable input device 
may cooperate with a receiver that receives signals 
from the input device. The remote controllable input 
device has a joy stick for moving a cursor on a display 
section, menu designating keys or buttons and ten keys. 

Memory drive 23 drives memory medium storing 
the computer program that is required to perform oper- 
ation of navigation processing unit 10 and agent 
processing unit 1 1 . The computer program stored in the 
memory medium includes various programs and data. 
The memory medium may be any one capable of stor- 
ing the computer program. For example, magnetic 
memory medium such as a floppy disk, hard disk and 
magnetic tape, semi-conductor memory medium such 
as a memory tip and IC card, magneto-optical memory 
medium such as CD-ROM, MO and PD, etc. may be 
examples of suitable memory medium. Any printed mat- 
ter describing the computer program may also used in 
cooperation with a character analyzer. 

When a rewritable memory medium such as a 
floppy disk and IC card is used, it may also be used to 
store data in RAM of navigation processing unit 10 and 
agent processing unit 11 or memory 29, 30. For exam- 
ple, study data (study item data and response data) 
regarding activities of the agent is stored in an IC card. 
The driver may use the IC card storing the driver's own 
data in common for two or more vehicles, as far as these 
vehicles have the same device. This means that the 
agent is pecutiar not to the vehicle but to the driver. 

Communication control device 24 transmits and 
receives data to and from a portable telephone and per- 
mits communication between the driver and his or her 
friend outside of the vehicle. Communication control 
device 24 also receives data regarding road information 
including traffic jams and traffic accident information 
that are supplied from a traffic information center. It also 
receives Karaok^ data. The study data regarding activi- 
ties of the agent may be received and transmitted 
through communication control device 24. 

Voice output device 25 comprises a plurality of 
speakers arranged in the vehicle through which a voice 
controlled by voice control unit 14 is outputted. The out- 
putted voice includes a voice message for guidance of 
the drive route and the agent's voice or sound that is a 
kind of the agent'6 activities. The speakers for audio 
instruments may be used as voice output device 25. 
Voice control unit 1 4 changes accent and tone quality of 
the voice outputted by voice output device 25 in 
response to the driver's tuning command. 

A microphone 26 functions as voice input device for 
inputting any voice which is then processed by voice 
control unit 14. For example, a voice for inputting the 
goal of the drive route in navigation processing opera- 
tion and the driver's voice in response to the agent's 



activities are inputted through microphone 26. A micro- 
phone for Karaoke may be used as microphone 26. 
However, microphone 26 should preferably have direc- 
tionality enough to collect the driver's voice. A hand-free 
5 unit comprising speaker 25 and microphone 26 may be 
used for telecommunication. 

Display 27 is used to represent the results of oper- 
ation by navigation processing unit 10, for example, 
guidance to operation, operation menu, operation keys, 

io a recommended drive route to the goal determined by 
the user's input through input device 22, and a guide 
map along the drive route. Display 27 also represents 
the results of operation by agent processing unit 1 1 , for 
example, the activities of the agent. Any image taken by 

75 camera 28 is also shown on display 27 after being proc- 
essed by image processing unit 13. As display 27 may 
be used a CRT display, a liquid crystal display, a plasma 
display or a hologram device that projects a hologram 
onto a front glass. 

20 Camera 28 comprises CCD cameras that take pho- 
tographs inside and outside of the vehicle. An inside 
CCD camera mounted in the vehicle take photographs 
of the driver. CCD cameras are also mounted at the 
front, rear, right side and left side for taking photographs 

25 of surroundings of the vehicle. The photographs taken 
by camera 28 are supplied to image processing unit 13 
for image analyzing operation. The result of the image 
analyzing operation by image processing unit 13 is used 
to determine a program number by agent processing 

30 unit 1 1 . 

Agent data memory 29 stores data and program 
necessary for agent processing operation in accord- 
ance with this embodiment of the present invention. 
Memory 29 comprises memory medium such as a 

35 floppy disk, hard disk, CD-ROM, magneto-optical disk, 
magnetic tape, IC card, magneto-optical card, etc. and 
a drive for the memory medium used. In a preferred 
embodiment, study item data 292 and response data 
293 are stored in a portable memory medium such as 

40 IC card and floppy disk, and other data is stored in a 
hard disk. 

Agent data memory 29 stores agent program 290. 
program select table 291, study item data 292, 
response data 293 and image data 294. Study item data 

45 292 and response data 293 represents the result of 
study of the agent through the driver's operation and/or 
response. Accordingly, study item data 292 and 
response data 293 are peculiar to each driver and 
renewed when the driver makes some operation and/or 

so response. 

Image data 294 stores the agent appearing on dis- 
play 27. As described before, the agent represented on 
display 27 may be like a human body, like an animal 
such as a chick, dog, cat, frog, rat, etc., robot or any 

55 imaginary character. The agent may be a child at the 
beginning, which is grown with time as a result of study. 
Image data 294 stores a plurality of the agent appear- 
ance, one of which may be selected by the driver 



5 



BNSDOCID: <EP 



O893308A2.J_> 



9 



EP 0 893 308 A2 



10 



through input device 22. 

Agent program 290 stores agent processing pro- 
gram and communication program in numerical order. 
Agent processing program commands activities of the 
agent. Communication program commands communi- s 
cation between the agent and the driver which is output- 
ted through display 27 and speaker 25. 

Agent program 290 stores plural kinds of voice pat- 
terns, for example, man's voice, woman's voice, child's 
voice, mechanical sound, animal voice, famous actor's u 
or actress's voice, and popular character's voice. The 
driver can select favorite one of the voice patterns 
stored in agent program 290 through input device 22. 

Program select table 291 is a table for use in selec- 
tion of a specific one from communication programs is 
stored in agent program 290. An example of program 
select table 291 is shown in Fig. 2. Once a specific pro- 
gram is selected in reference to program select table 
291 in Fig. 2, the same program number is applied to an 
activity pattern select table, an example of which is 20 
shown in Fig. 3, to determine the activity (appearance 
and voice) of the agent. The program number of tables 
in Figs. 2 and 3 are identical to the communication pro- 
gram number stored in agent program 290. 

Figs. 4A-4D show an example of the agent activi- 25 
ties, which is shown on display 27 when one of the pro- 
gram Nos, 00001-00006 is selected (see Fig. 3). This 
activity start with the image of Fig. 4A which is changed 
to the images of Figs. 4B and 4C, then to the image of 
Fig. 4D that is the same as the starting image of Fig. 4A. 30 
This activity is a respectful and ceremonious bow, 
because this is almost the first time that Lisa meets the 
driver. The name "Lisa" is only an example and any 
name may be inputted by the driver. With the agent's 
action of obeisance appearing on display 27, the agents 35 
says something that the driver can hear through 
speaker 25. In the program No. 00001 or 00002, the 
agent makes an obeisance while saying "Nice to meet 
you, I'm Lisa. I wish to go with you", which is, however, 
changed depending upon the vehicle's and driver's cur- 40 
rent status and as a result of the past record study. 

In this embodiment, input device is also used to 
input the user-related information 298 to be stored in 
agent data memoiy )29a. The user-related information 
298 may include th^tiser's a g 6i seXp hobby, character, 45 
favorite regarding thfe agent, etc. The user-related infor- 
mation 298 stored in agent data memory 29a is read out 
by agent processing unit 1 1 to determine the personal 
characteristic of the agent. The user-related information 
298 is also used to determine utterance pattern of the so 
agent to be outputted through voice output device or 
speaker 25. The user-related information 298 may be 
obtained from the user's response to the agent's ques- 
tion, which is inputted through microphone 25. 

When current status sensor 40 detects that the ss 
coolant temperature is lower than a predetermined 
level, the agent shows a sleepy activity. For example, 
the agent gives or stifle a yawn before making a prede- 



termined action (e.g., bow). In another example, the 
agent shows a slower movement and speaking. Such 
sleepy expression may be changed as a result of the 
study. A cheerful activity may be expressed by the agent 
who speaks loudly, appears on display while running, 
for example. 

The respective items shown on the abscissa of 
table 291 of Fig. 2 determine the program number, on 
the ordinate, to be selected. The items on the abscissa 
may be classified into two groups, one being deter- 
mined from the vehicle or driver's condition detected by 
current status sensor 40 which includes time, vehicle 
location where ignited, coolant temperature, etc. and 
the other being determined from the study results 
stored in study item data 292 and response data 293 
which includes the number of today's ignition-on opera- 
tion, time passage from the last ignition-off operation, 
the total number of ignition, etc. In table 291 of Fig. 2, a 
circle indicates the item that is indispensable to select 
the program of a corresponding number. A hyphened 
item is not considered in selecting the program of a cor- 
responding number. 

Although Figs. 2 and 3 describes the activity pat- 
tern (bow) to be represented in response to the igniting 
operation, various activity patterns are likewise defined 
in response to some operation. For example, push of a 
brake pedal to an extent greater than a predetermined 
level is regarded as a key operation, in response to 
which the agent may fall on his or her behind, take sev- 
eral steps backward, keep his or her foot, cries in sur- 
prise, etc. These responsive activities may be changed 
by the agent's study of sudden braking operation. For 
example, the program determines that the agent fells on 
his or her behind at the first to third times of sudden 
braking operation, takes several steps backward at the 
fourth to tenth times, and keeps his or her foot thereaf- 
ter. Such program represents that the agent becomes 
experienced and accustomed to the driver's sudden 
braking operation. It may be that when a time of more 
than one week has been passed since the last time sud- 
den braking operation, the agent's activity of one earlier 
stage is applied. For example, where the fifth time sud- 
den braking operation occurs ten days after the last time 
(the fourth time) operation, the program determines 
such that the agent falls on his or her behind. 

Study item data 292 and response 293 are stored 
and renewed by the agent's study, examples of which 
are shown in Figs. 5 and 6, respectively. Study item data 
292 shown in Fig. 5 includes the total number of igni- 
tion-on, the last ignition-off, today's ignition-on, the fuel 
remainder record, the skip record, the default value, etc. 

The total number of ignition-on operation is 
increased by one each time the ignition switch is turned 
on to ignite the engine. Each time the ignition switch is 
turned off, its date and time is renewed at the last igni- 
tion-off data. Each time the ignition-on operation occurs, 
its number (in a day) and its ignition-off time are 
recorded in today's ignition-on data. When a day is over 
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(at 24:00), this data is cleared to zero, awaiting the next 
ignition-on operation. The time for renewing the today's 
ignition-on operation data is normally 24:00, but may be 
changed upon the driver's demand. 

The fuel remainder record stores fuel remainders at s 
the last five times fill-up. Each time the fuel is refilled, 
the newest data is stored at the rightmost column, and 
the last four data are moved to the adjacent left col- 
umns, respectively. The oldest data in the leftmost col- 
umn is deleted. From the last five fuel remainders may io 
be obtained the average fuel remainder (G2). When the 
fuel sensor 415 detects the current fuel remainder (Q1) 
is below the average fuel remainder (G2), the agent 
appears on display 27 to show a fill-up suggesting activ- 
ity and says Tve got hungry. I want fuel." through is 
speaker 25. 

The skip record data is counted up each time the 
driver refuses or neglects the agent's recommendation 
or communication even when such program number 
has been selected under control of the present inven- so 
tion. The study item data 292 includes the skip record 
where the agent activities of the program number 
involves the driver's response, such as by Program No. 
00123 wherein the agent suggests stop of operation of 
the air-conditioner. 25 

The default value includes initial values such as 
time, number temperature, vehicle speed, data, etc. 
The default value is used to initialize data when data 
has been changed by the driver, for example, when the 
driver has changed the ignition-on operation time from 30 
the initial value of 24:00 to his or her desired time, as 
described before. 

The other data that may be included in the study 
item data 292 include, for example, birthdays of the 
driver and his or her relatives and friends (that can be ss 
inputted by the driver), national holidays and their histo- 
ries, date of events such as Christmas Eve, St. Valen- 
tine Day, etc. Among the communication programs, 
there may be special menu for special day. In accord- 
ance with an example of the special menu communica- 40 
tion program, the agent is appearing on display 27 while 
dressed as ; Santa Claus on Christmas Eve and Day. 

The response data 293 in Fig. 6 stores the user's 
response histbry<to the agent's activities for the commu- 
nication programs which include the user's response as as 
one of the study data. The response data have different 
contents for different communication programs. In an 
example shown in Fig. 6A, the response data for the 
programs of Nos. 00123 and 00125 store several latest 
data (time and content), The response data of Program so 
No. 00124 stores the only one latest response content, 
which is renewed each time the driver makes a 
response. In other examples, the response data may be 
several latest response contents (not including 
response time), only one latest response data of time ss 
and content, only one or several latest response date 
(not including response contents, etc. 

The symbols A, B and C in Fig. 6A indicate the 



response contents. As shown in Fig. 6B, the symbol A 
indicates "Neglected", B "Refused" and C "Accepted". 
The driver's response content may be discriminated by 
analyze of the driver's voice inputted through micro- 
phone 26 or of information inputted through input device 
22. 

Although in this embodiment the driver's response 
contents are classified into three patterns, "Neglected" 
(A), "Refused" (B) and "Accepted" (C), it may have more 
detailed classification. For example, "Resolutely 
Refused", "Angrily Refused", "Pleased", etc. may be 
added. 

Fig. 7 represents data files stored in navigation data 
memory 30 (Fig. 1). As shown in Fig. 7, navigation data 
memory 30 stores communication area data file 301, 
map data file 302, crossing data file 303, node data file 
304, road data file 305, route-finding data file 306 and 
photo data file 307. Navigation data memory 30 com- 
prises one or combination of storage medium including 
a floppy disk, hard disk, CD-ROM, magneto-optical disk, 
magnetic tape, IC card, magneto-optical card, etc., and 
drive units therefor. For example, route-finding data file 
46 is made of rewritable memory such as flush memory 
while other data files are stored in CD-ROM, and drive 
units therefor are respectively used. 

Communication area data file 301 stores communi- 
cation are data per handy phone types. Communication 
data is used for representation, on display 27, of com- 
municatable area by the driver's handy phone con- 
nected or non-connected to communication control unit 
24. This data is also used while finding out the drive 
route to the goal. The respective communication area 
data is assigned its peculiar number for easy access 
thereto. The communicative area is defined by continu- 
ity of short segments, so that the communicative area 
data may be specified by a plurality of coordinate data 
representing points connecting the segments. In 
another example, the communicative area is divided 
into plural rectangles, each rectangles bang defined by 
two diagonal points thereof, in which case the communi- 
cative area data comprises a plurality of coordinate data 
representing diagonal points of the respective rectan- 
gles in the communicative area. 

Data in communication area data file 301 is prefer- 
ably renewable to conform with change of the communi- 
cative area of the handy phone. To meet this 
requirement, the handy phone is connected to commu- 
nication control unit 24 for communication with an infor- 
mation center for updating of the contents of 
communication area data file 301. In another example, 
communication area data file 301 is made of rewritable 
and updatable memory such as a floppy disk and IC 
card. 

Map data file 302 stores map data to be repre- 
sented on display 27. Map data is stored in a tree con- 
struction. The uppermost layer stores map data of the 
broadest territory, and lower layer stores more detailed 
map data of smaller territories. The map data of the 
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respective layers are assigned their peculiar map 
codes. 

Crossing data file 303 stores crossing data such as 
crossing numbers specifying the crossing one by one, 
crossing names, crossing coordinates, road specifying s 
numbers start or terminate with the crossings, existence 
of signals at the crossings, etc. 

Node data file 304 stores node data specifying 
coordinates of respective points on the respective 
roads. Any road section may be defined by a plurality of 10 
node data thereon between the beginning and end of 
the section. 

Road data file 305 stores road numbers specifying 
the road one by one, the crossing numbers at the begin- 
ning and end of the respective roads, other road num- 15 
bers of the road that begins or terminates with the same 
point, road width, prohibition information (no parking, no 
entrance, one-way road, etc.), number of photo data, 
etc. 

Data stored in crossing data file 303, node data file so 
304 and road data file 305 are particularly used for 
route-finding. 

Route-finding data file 306 stores a series of cross- 
ing data and a series of node data which constitute the 
route found in the navigation system. A series of cross- 25 
ing data includes data regarding names and numbers of 
the crossings on the suggested drive route, numbers of 
the photos showing these crossings, names and num- 
bers of the crossings at which the vehicle should turn to 
follow the suggested drive route, distances between the 30 
crossings, etc. A series of node data includes data 
regarding coordinates of the respective nodes on the 
suggested drive route. 

Photo data file 307 stores photos showing the 
crossings, conspicuous landscape and buildings, for 35 
example, along the suggested drive route. Photos are 
assigned their peculiar photo numbers and stored in the 
form of digital, analog or negative films. 

As shown in Fig. 8, current status sensor 40 
includes an ignition sensor 401 , a vehicle speed sensor 40 
402, an accelerator sensor 403, a brake sensor 404, a 
side brake serisbr 4o5, a shift position sensor 406, a 
winker sensor 407; ja wiper sensor 408, a light sensor 
409, a seat-belt sensor 410, a door opening sensor 41 1 , 
a passenger sensqh412, a room temperature sensor 45 
413, an open airterfiperature sensor 41 4, a fuel remain- 
der sensor 415, a coolant temperature sensor 416, an 
ABS sensor 417, an air-conditioner sensor 418, a 
weight sensor 419, a fore-distance 6ensor 420, a rear- 
distance sensor 421 , a body temperature sensor 422, a so 
pulse rate sensor 423, a perspiration sensor 424, a 
brain wave sensor 425, an eye-tracer 426, an infrared 
sensor 427. and other sensors 428 for detecting, for 
example, decrease of air pressure of tires, loosening of 
belts, opening of windows, operation of horn, oil temper- ss 
ature, oil pressure, etc. As noted from the above, cur- 
rent status sensor 40 is used to detect various vehicle 
conditions and driver's conditions in this embodiment. In 



modification, only vehicle conditions or only driver's 
conditions may be detected by current status sensor 40. 
Sensors are mounted at suitable location inside or out- 
side of the vehicle, respectively. Usually one sensor is 
used for one sensing purpose. However, it may be pos- 
sible that some kind of sensor receives the detection 
signal from another sensor to carry out indirect sensing. 
For example, an air-pressure sensor may indirectly 
detect a decrease of an air-pressure of a tire in 
response to a change of a signal from a wheel velocity 
sensor. 

Ignition sensor 401 detects ON and OFF of an igni- 
tion starter. Vehicle speed sensor 402 may be of any 
conventional type including one wherein a rotation 
angle velocity or revolution of a speed meter cable is 
detected to calculate a vehicle speed. Accelerator sen- 
sor 403 detects a degree of push stroke of an accelera- 
tor pedal. A brake sensor 404 detects a push stroke, 
push force or push speed of a brake pedal to discrimi- 
nate if the driver makes an emergency brake. Side 
brake sensor 405 detects operation or non-operation of 
a side brake. Shift position sensor 406 detects the cur- 
rent position of a transmission shift lever. Winker sensor 

407 detects which winker is lightened. Wiper sensor 

408 detects wiper drive conditions (especially, wiper 
speed). Light sensor 409 detects various lights and 
lamps such as head lights, tail lamps, fog lamps, room 
lamps, etc. being lightened. Seat-belt sensor 410 
detects if the driver and passengers are equipped with 
seat-belts. If at least one of them is not equipped with a 
seat-belt, the corresponding communication program is 
working such that the agent appears on display 27 to 
give warning, notice, advice, etc., one of which is 
selected by the results of study. 

Door opening sensor 411 detects opening of vehi- 
cle doors. When detecting a half opening of the door, it 
is informed to the driver or passenger by the agent's 
activities (action and voice). Door opening sensor 411 
may comprise plural sensors each detecting opening of 
doors at the driver's seat, at the assistant driver's seat, 
at opposite rear seats, etc., respectively. Passenger 
sensor 412 detects if there are any passengers in the 
vehicle, by the photo showing the interior of the vehicle 
taken by camera 28 or by pressure sensor or weight 
sensor mounted under the seat. Room temperature 
sensor 413 detects a room temperature of the vehicle, 
and open air temperature sensor 414 detects a temper- 
ature outside of the vehicle. 

Fuel remainder sensor 41 5 detects a remainder in a 
fuel tank. As described before, the communication pro- 
gram is selected when the fuel remainder is decreased 
to the average of the last five records, so that the agent 
urges the driver to refill the fuel. 

Coolant temperature sensor 416 detects a coolant 
temperature. When detecting that the coolant tempera- 
ture is lower than a predetermined level at the time 
immediately after the ignition switch is turned on, the 
agent usually takes a sleepy action. On the other hand, 
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when the coolant temperature is higher than another 
predetermined level, such communication program is 
selected that the agent looks weary and gives warning 
or notice to the driver to prevent overheating of the 
engine. 5 

ABS sensor 417 detects if an ABS (anti-braking 
system) is in operative. As well known, ABS prevents 
the tires from being locked due to emergency braking to 
thereby improve drivablility and stability of the vehicle. 

Air-conditioner sensor 41 8 detects the conditions of 10 
an air-conditioner mounted in the vehicle, including ON 
and OFF thereof, the set temperature and the set wind 
volume, etc. Weight sensor 419 detects the driver's 
weight, which is used solely or in combination with the 
photos by camera 28 to specify the driver so that the is 
agent's activities can be determined in accordance with 
study item data 292 and response data 293 inherent to 
the said driver. When there are different agents selecta- 
ble by plural drivers, one specific agent is determined by 
identification of the current driver. 20 

Fore-distance sensor 420 detects a distance to 
another vehicle or obstacles ahead of the vehicle cur- 
rent position on the road, whereas rear-distance sensor 
421 detects a distance to another vehicle or obstacles 
astern of the vehicle current position on the road. 25 

Body temperature sensor 422, pulse rate sensor 
423 and perspiration sensor 424 detects a body temper- 
ature, a pulse rate and perspiration of the driver, respec- 
tively. These sensors may be mounted on the surface of 
a steering to detect conditions of the driver's hand. Body 30 
temperature sensor 422 may be a thermography utiliz- 
ing infrared sensor elements to detect the driver's body 
temperature distribution. Brain wave sensor 425 detects 
alpha wave or beta wave of the driver to discriminate the 
degree of awakening of the driver. Eye-tracer traces eye 35 
movement of the driver to the driver's intention and 
awakening. Infrared sensor 427 detects movement of 
the user's hands and head. 

In reference to the flowchart of Fig. 9 showing the 
main routine of the agent control operation of the device 40 
of this embodiment, agent processing unit 1 1 first initial- 
izes the device after ignition sensor 401 detects that the 
ignition switch turned on, at step S11. Initialization 
includes clearing of RAM, allocation of the respective 
working areas i if* RAM, loading of program select table 45 
291 (Fig. 2) into^RAM, set of flags to zero, etc. Although 
the operation starts with turning of the ignition switch in 
this embodiment, it may start in response to detection of 
opening or closure of any door by door opening sensor 
411. so 

Next, agent processing unit 1 1 receives the current 
status data, stores them in their predetermined areas in 
RAM and analyzes them to confirm the current vehicle 
and user's status, at step S12. The current status data 
include data detected by the various sensors in current 55 
status sensor 40 and supplied therefrom to agent 
processing unit 1 1 through current status data process- 
ing section 15. The image data obtained by camera 28 
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is processed by image data processing section 13, 
which is also inputted to agent processing unit 11. The 
current vehicle position data detected by current vehicle 
position sensor 21 is also inputted to agent processing 
unit 1 1 through navigation processing unit 10. By way of 
example of the current status data, where coolant.tem- 
perature sensor 416 detects a coolant temperature tl , 
agent processing unit 1 1 stores this temperature tl in a 
predetermined area in RAM and, if the temperature tl is 
below a predetermined temperature t2, discriminates 
that the coolant temperature is low (see the coolant 
temperature column in table of Fig. 2). 

When the driver inputs any request through micro- 
phone 26, this is processed by agent processing unit 1 1 
as a kind of the current status data. Such driver's 
request may be inputted by saying "Phone to Mr. Thin- 
gummy", "Show me restraints around here", "Play a 
CD", etc. The driver's request is processed by voice 
processing unit 14 with keywords such as "phone", 
"restraint", "play", "CD". These keywords are arranged 
on the abscissa of program select table 291 in Fig. 2. 

Agent processing unit 1 1 then checked study data 
(study item data 292 and response data 293) in agent 
data memory 29 to confirm the agent's study regarding 
the obtained current status data, at step S13. 

From the obtained current status data and the study 
data thereof, agent processing unit 1 1 discriminates if 
there is any communication program to be initiated, in 
reference to program select table 291 , at step S1 4. If not 
(No at S14), the procedure is returned to step S12 
where it awaits another current status data. Where 
there is a communication program to be initiated (Yes at 
S14), it confirms the program number thereof. It further 
reads out the driver's response record stored in 
response data 293 for the said program number to 
thereby determine if the communication program of the 
selected program number should be initiated or sus- 
pended, at step S15. 

If the answer at Si 5 is No, agent processing unit 1 1 
operates to initiate the selected communication pro- 
gram so that the predetermined agent's activities are 
shown on display 27 and outputted through speaker 25. 
at step S1 6. This enables good communication 
between the driver and the agent which reflects not only 
the vehicle and driver's current status but also the result 
of study of the past status data and the driver's 
response records. 

When the driver shows any response to the agent's 
activities made at S16, such response data is obtained 
through microphone 26 or input device 22, at step S1 7. 

Then, agent processing unit 1 1 discriminates if the 
communication program made at S16 involves any sug- 
gestion, proposal or question that awaits the driver's 
response or answer. For example, agent says "Do you 
want to hear any radio program ?" or "Do you feel hun- 
gry? May I find any restraint near here?". The latter pro- 
gram may be initiated in response to detection 
hungry sound through microphone 26. Such a V 
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communication program is hereinlater referred to as a 
"questioning program". 

When the selected communication program is not a 
questioning program (No at S18), the procedure is 
advanced to a step S20. When it is a questioning pro* s 
gram (Yet at S18), at a step S19 ( agent processing unit 
1 1 controls the agent's activities in accordance with the 
driver's response already obtained at S1 7. For example, 
when the driver indicates the intention to accept the 
agent's proposal "Do you want to hear any radio pro- 10 
gram V by answering "Yes", "OK", "Please", etc., agent 
processing unit 1 1 urges the agent to act accordingly 
and turns the radio receiver on to follow the driver's 
request. 

Agent processing unit 1 1 accumulates data regard- is 
ing the execution of the selected communication pro- 
gram to obtain the study data, at step S20. For example, 
when the selected communication program is not exe- 
cuted (Yes at S15), the number of skip of this program 
number stored in study item data 292 is increased by 20 
one. When the current number of skip is lower by one 
than a predetermined maximum number, ail records 
now stored in study item data 292 and response data 
293 for that program number should be cleared. Except 
this case, the current record in the study item data 292 25 
is renewed when the current status data acquired at 
Step 12 includes some study item, and the response 
data obtained at Step 1 7 is stored in the response data 
293 when the selected communication program is a 
questioning program that requests the driver's 30 
response. When a predetermined number of response 
records have already been stored in the response data 
293, the oldest response data is disused. 

After executing the above-described steps, at step 
S21, agent processing unit 11 confirm that a flag has 35 
become one in a flag area that was set in RAM at Step 
11. When the flag is still zero (No at S21), it then dis- 
criminates if ignition sensor 401 detects that the ignition 
switch is turned off, at step S22). When not detected 
(No at S22), the procedure is returned to S12. When 40 
turning-off of the ignition switch is detected (Yes at S22), 
the flag in the flag &rea in RAM is set to one, at step 
S23, and then the procedure is returned to S12. When 
the flag is set to onjd, agent processing unit 1 1 controls 
such that the agep&takes a farewell action that is the 45 
final activity in the Selected communication program (at 
S16), and obtains the study data (at S20). It also exe- 
cute the automatic extinguish any light that the driver 
has forgotten to turn off. Then, the control procedure 
terminates (Yes at S21 ). so 

Figs. 10A-10C show an example of the agent activ- 
ities that is controlled by agent processing unit 11 after 
the ignition switch is turned on. In this example, ignition 
sensor 401 detects that the ignition switch is on, current 
position sensor 21 detects the current vehicle position is ss 
other than the driver's home and rest positions. These 
data are supplied to agent processing unit 1 1 as the cur- 
rent status data (at S1 2). Regarding study item data 292 



and response data 293, it confirms that this is the sec- 
ond ignition -on operation today, the last agent control 
operation was terminated at 10:05, on July 12, 1997 t 
and the total number of ignition-on operation amounts to 
30 times, at S13. 

From these data, agent processing unit 1 1 deter- 
mines the communication program of No. 00004 should 
be executed, in reference to the program select table 
291 shown in Fig. 2 (at S14). The agent activity to be 
outputted through display 27 and voice output device 25 
is an "obeisance" that is a respectful bow, with saying 
"Hello, I'm Lisa. I will go with you again". After confirm- 
ing that this communication program should not be sus- 
pended (No at S15), agent processing unit 1 1 starts this 
communication program (at S16), as shown in Fig. 10B. 
The agent's obeisance is represented by a series of sta- 
tionary images or animation. 

Since the program of No. 00004 is not a questioning 
program that awaits the driver's response (No at S18), 
the study data is stored and renewed as a result of exe- 
cution of the program 'of No. 00004 (at S20). In this 
example, the number of today's ignition-on operation is 
changed from the previous value "2" to "3", and the total 
number of ignition-on operation is changed from the 
previous value "30" to "31". Since the flag is still zero at 
this time (No at S21) and the ignition is still on (No at 
S22), the procedure is returned to S12 so that this com- 
munication program continues. 

Since the total number of ignition-on operation has 
reached "31", the communication programs of Nos. 
00001-00006 can not be selected (see table 291 in Fig. 
2), as far as all study data (study item data 292 and 
response data 293) are initialized. Until the total number 
of ignition-on operation reaches "300", one of the pro- 
grams of Nos. 00007-00019 is selected in response to 
the current status data. The agent no more makes an 
"obeisance". The agent makes an ordinary "bow" when 
the coolant temperature is high, while a "sleepy bow" is 
applied when the coolant temperature is low. The 
agent's voice depends on the program number to be 
selected by data such as the place where the ignition 
switch is turned on. the number of today's ignition-on 
operation, etc. 

Figs. 11A-11C illustrate another example of the 
agent's activities that may be selected during driving. In 
this example, air-conditioner sensor 418 detects that 
the air-conditioner is in operation, and temperature sen- 
sors 413, 414 detects a room temperature T1 and an 
open air temperature T2, respectively. These data are 
supplied to agent processing unit 1 1 as the current sta- 
tus data. Agent processing unit 11 investigates the 
response data 293 to find that the driver's last twice 
response to the agent's activities of the program No. 
001 23 are both "C" indicating that the driver accepts the 
agent's suggestion or advice (see Fig. 6A), and reads 
out the default values 0*3*2, T4=24) from study item 
data 292. From the default values T3, T4, it is confirmed 
that the temperature relationships of T1 -T2 ^ T3 and T2 
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£ T4 are satisfied (see Fig. 1 1 A). 

Agent processing unit 1 1 thus selects the communi- 
cation program of No. 001 23 in reference to the program 
select table 291 (at S14). After confirming that this pro- 
gram is not a questioning program that awaits the s 
driver's response (No at S15), it starts executing this 
program (at S16) so that the agent's activities such as 
shown in Fig. 1 1 B will be outputted through display 27 
and speaker 25. More particularly, the agent appears 
with smile on display 27 and says "It becomes cool. May 10 
I suggest introducing the open air?". 

Agent processing unit 1 1 then acquires the driver's 
response to the agent's activities (at S17). Since the 
program No. 001 23 is a questioning program that awaits 
the driver's response (Yes at S18), it controls operation 75 
of the air-conditioner to follow the driver's request. 
When the driver's response is "acceptance", the air- 
conditioner is turned off and the front side windows are 
half -opened (at S19). 

After executing control to follow the driver's 20 
response, agent processing unit 1 1 stores a result of 
study depending upon the driver's response pattern. 
Examples of the study are shown in Fig. 11 C. In this 
case, when the driver neglects the agent's suggestion 
(response pattern A), application of the program No. 25 
00123 is suspended one time. Moreover, the tempera- 
ture default values T3 and T4 to be used in the next 
application of the program No. 00123 are respectively 
decreased by one, so that the next T3 should be 1 °C 
and the next T4 should be 23°C. When the driver 30 
refuses the agent's suggestion (response pattern B), 
application of this program is suspended five times, and 
the temperature default values T3 and T4 to be used in 
the next application of this program are respectively 
decreased by one, so that the next T3 should be 1°C 35 
and the next T4 should be 23°C. When the driver 
accepts the agent's suggestion, the current default val- 
ues 73, T4 remains unchanged, and this program is not 
suspended but executed when selected. 

Although in the above description the front side win- 40 
dows are half -oppned in response to the driver's accept- 
ance to the agent's suggestion, it may be possible that 
agent processing unit 1 1 performs more detailed control 
for opening thje jwindows as a result of study. More par- 
ticularly, data regarding window position and degree of 45 
window opening are also subjected to the agent's study. 
For example, the driver may say "Too much opened", 
"Close a little bit more", "Open rear side windows too", 
etc. after half-opening the front side windows in 
response to the driver's acceptance to the agent's sug- so 
gestion. Such response reflects the driver's favorite con- 
dition regarding the window opening and, therefore, are 
stored in the response data 293. The default values for 
window opening may be, for example. 1/2 for front side 
windows. When the driver says "Open rear side win- ss 
dows too", this is regarded as a kind of the current sta- 
tus data (at S12) so that a corresponding 
communication program is executed (at S16) to open 



the rear windows, with the corresponding agent s activi- 
ties appearing on display 27 and outputted through 
speaker 25. 

In accordance with a modified embodiment of the 
present invention shown in Figs. 12-16, the agent is 
more personified by selectively controlling the agent's 
voice output pattern. In this embodiment, even if the 
same communication program is selected, the agent 
talks to the driver in a different voice pattern or tone, 
depending on the vehicle condition, the user's favorite to 
the agent, the weather condition, etc. The agent's activ- 
ity or voice pattern may be determined by also taking his 
or her character into consideration. 

In this embodiment, agent data memory 29 in Fig. 1 
is replaced with another one 29a that stores voice data 
295, agent's character select table 296, voice pattern 
select table 297 and user-related information 298, in 
addition to agent program 290, program select table 
291 , study item data 292, response data 293 and image 
data 294, as shown in Fig. 12. Except for difference in 
contents of agent data memories 29 and 29a, the 
device is almost similar to that shown in Fig. 1 and 
therefore no description of the other parts and elements 
of the device is given here to avoid tedious repetition. 

Agent data memory 29a in this embodiment com- 
prises one or plural memory medium that can be 
selected from floppy disk, hard disk, CD-ROM, mag- 
neto-optical disk, magnetic tape, ICcard, magneto-opti- 
cal card, etc. and drive unit for the memory medium 
used. It may be preferable that study item data 292, 
response data 293 and user-related data 298 are stored 
in removable memory medium such as floppy disk and 
IC card and other data are stored in a hard disk. 

Voice data 295 is to output the agent's communica- 
tion through speaker 25. A plurality of predetermined 
words and clauses stored in the voice data 295 may be 
combined upon demand to synthesize the agent's com- 
munication. The driver or user may select appearance 
and voice of the agent through input device 22. The 
agent's voice may be selected, lor example, from man's 
voice, woman's voice, child's voice, mechanical voice, 
animal voice, famous actor's or actress's voice, etc. 
Once selected appearance and voice of the agent may 
be changed at any time the user so wishes. When some 
voice pattern is selected, it can be outputted through 
speaker 25 after being tuned by voice control unit 14 in 
accordance with the selected voice pattern. 

Agent's character select table 296 is used to deter- 
mine the agent's character in accordance with the user- 
related information 298. The agent's character selected 
in reference to the table 296 is used as data for deter- 
mining the voice pattern. 

An example of the agent's character select table 
296 is shown in Fig. 13. As shown in the rightmost col- 
umn of the table 296 in Fig. 13, the agent's character is 
determined as one selected from the group consisting 
of "obliging", "moderate", "crybaby", "strong sense of 
justice", "beautiful secretary", "competent manager", 
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"high-school girl", etc. When a "beautiful secretary- 
agent is selected, her speaking tends to be placid. 
When a "high-school girl" is selected, she usually 
speaks cheerfully but soon become displeased when 
the vehicle is in a traffic jam or exposed to the weather. 5 
Of course, these are only examples. 

The agent's character is selected based on the 
vehicle condition such as the vehicle type and the 
number of ignition-on operation, the users own data 
such as the user's age, sex, hobby and character, the 10 
current weather condition, the current time, etc. These 
items may be inputted by the user through input device 
22 and/or acquired from the user's response (inputted 
through microphone 26) to the agent s question (output- 
ted through speaker 25). Some of the items may be is 
automatically stored in the table 296 by the driver's vehi- 
cle operation and/or signals from sensors. Among these 
items thus obtained, the user's own data is also stored 
in a memory section of the user-related information 298 
in agent data memory 29a. so 

The voice pattern select table 297 is to determine 
the voice output pattern for tuning of the agent's voice. 
An example of the voice pattern select table is shown in 
Fig. 14. As shown in the rightmost column of the table 
297 in Fig. 14, the agent's voice pattern is determined 25 
as one selected from the group consisting of "loud", 
"placid", "cool", "respectful", "tender", "kind", "trem- 
bling", "angry", "reluctant", "cheerful", "out of humor", 
etc. The voice pattern is selected, in reference to the 
table of Fig. 14, from the type of the agent's communi- 30 
cation, the vehicle condition, the agent's character, the 
user's favorite, the external factors, the current time, etc. 

In this example, the agent's communication type on 
the abscissa of the table 297 in Ftg. 14 is classified into 
at least A-F types, A means route-guidance, B warning 35 
message. C explanation, D greetings, E call-back, and 
F conversation. The agent's communication type 
depends on a specific communication program to be 
selected in reference to the program select table 291 of 
Fig. 2. In other words, each communication program 40 
has its data regarding the communication type, that is 
one of the abbve-jdefecribed types A-F. 

The vehicle condition on the abscissa of the table 
297 in Fig, 14 incjUcjIes whether or not the vehicle is in a 
high-speed (say, (n^ than 80km/hour) drive, whether 45 
or not the vehicle 'is out of the prescheduled route, 
whether or not the vehicle interior is noisy (than a prede- 
termined noise level), the total number of ignition-on 
operation, etc. 

The agent s character on the abscissa of the table so 
297 in Fig. 14 is determined in reference to the table of 
Fig. 13, in such a manner as described above. 

The favorite on the abscissa of the table 297 in Fig. 
14 is selected by the user's input through input device 
22 or inputted from the user's response to the agent's ss 
question. The driver's favorite may be classified into 
"young lady", "placid lady", "showy lady", "cheerful girl", 
etc. 



The external factors also on the abscissa of the 
table 297 in Fig. 14 include the current weather condi- 
tion (fine, cloudy, rain, snow, etc.), whether or not the 
vehicle is in a traffic jam, whether or not the vehicle is 
driving on an uphill or downhill, etc. These external fac- 
tors may be automatically inputted by detection of cur- 
rent status sensor 40. 

The current time also on the abscissa of the table 
297 in Fig. 1 4 indicates the current time is morning (e.g. , 
from the sunrise to noon), daytime (e.g., from noon to 
the sunset), evening (e.g. ( from the sunset to 23:00) or 
midnight (e.g., from 23:00 to the next sunrise). The cur- 
rent time classification may be determined from detec- 
tion by a timer equipped in agent processing unit 1 1 and 
a calender (that also stores each day's anticipated sun- 
rise and sunset time) stored in agent data memory 29. 

When the voice output pattern is determined in ref- 
erence to the table 297, the agent's communication 
(speaking) is tuned accordingly by voice control unit 14. 
How to tune the agent's speaking in accordance with 
the determined voice output pattern is not limitative. For 
example, when the voice output pattern is determined 
as the loudly-speaking pattern, the agent's communica- 
tion is outputted through speaker 25 with an increased 
volume. In case of the placidly- or respectfully-speaking 
pattern, voice control unit 14 controls such that the 
agent speaks slowly so that the driver clearly under- 
stands the agent's communication word by word. The 
respectfully-speaking pattern may be tuned for giving a 
silent interval at each punctuation mark a little bit longer 
than by the placidly-speaking pattern. In case of the 
cheerfully-speaking pattern, the agent's communication 
is tuned such that the agent speaks in a cheerful and 
higher voice. The agent speaks in a manner that the 
ending of each word is pronounced in a lower voice. 

The user-related information 298 in agent data 
memory 29a stores the vehicle type of the user* posses- 
sion, the user's age, sex, character, favorite to the 
agent, etc. These user-related data is used to deter- 
mine not only the agent's character in reference to the 
table 296 of Fig. 13 but also the voice output pattern in 
reference to the table 297 of Fig. 14. The user-related 
data is stored for each driver and subject to the study. 

Fig. 15 is a flowchart showing a main routine of the 
agent control operation in accordance with this embodi- 
ment. Agent processing unit 1 1 first initializes the device 
after ignition sensor 401 detects that the ignition switch 
is turned on, at step S1 1 1 . This step is almost identical 
to S1 1 of the flowchart of Fig. 9, description of which is 
incorporated herein. 

Agent processing unit 1 1 then identifies the current 
driver, at step S1 12. More particularly, when the driver's 
voice is inputted through microphone 26. it is analyzed 
by voice control unit 14 to identify who is the current 
driver. The current driver may be identified by photo 
taken by camera 28 and then analyzed by image 
processing unit 19. The weight detected by weight sen- 
sor 419, the driver's seat position, the angle of the room 



12 

<EP 0893308A2 I > 



23 



EP 0 893 308 A2 



24 



mirror may also be taken into consideration to identify 
the current driver. In a preferred "embodiment, S12 
includes a driver-identifying communication program 
(not shown) in which the agent appears on display 27 
and speaks through speaker 25 for confirming that the 5 
current driver is just as identified. 

Once the current driver is identified and confirmed, 
agent processing unit 1 1 obtains and confirms the cur- 
rent status, at step S113. This step is almost identical to 
S1 2 of the flowchart of Fig. 9, description of which is 10 
incorporated herein. 

Then, at step S1 14, agent processing unit 11 per- 
forms the agent control operation depending on the cur- 
rent status confirmed at S113, in a manner to be 
described hereinlater in reference to the flowchart of is 
Fig. 16. As a result of the operation at S1 14, the agent 
makes some judgement, communication (action and 
speaking), control, study, inspection, etc. It may be pos- 
sible that the agent do nothing when so controlled by 
the current status. 20 

Then, agent processing unit 1 1 discriminates if the 
main control routine should be terminated, at step 
S115. If the main control routine should continue, the 
procedure is returned to S1 13 to repeat the operation 
through S1 1 3-S1 1 5. On the contrary, when ignition sen- 25 
sor 401 detects that the ignition switch is turned off (at 
S1 13). this means that the main control routine should 
be terminated (No at Si 15). In this case, after predeter- 
mined terminative operation such as turning-off of room 
interior lamps is made at S1 14, this routine of Fig. 15 is so 
terminated. 

Fig. 16 is a flowchart showing the agent control 
operation that is carried out in Si 1 4 of the main routine 
of Fig. 15. At step S121, agent processing unit 11 dis- 
criminates if there is any communication program to be 35 
initiated, in reference to program select table 291 . This 
step is almost identical to S14 of the flowchart of Fig. 9, 
description of which is incorporated herein. If there is no 
communication program to be applied (No at S121), this 
sub-routine is terminated and returned to the main rou- 40 
tine of Fig. 15. 

When there lis some communication program to be 
initiated (Yes at §121), it determines the agent's charac- 
ter by applying ;the current status (the number of igni- 
tion-on operatic^ current weather condition, time, etc.) 45 
that has been obtained at S113 of the main routine of 
Fig. 15 and the user-related information 298 in agent 
data memory 295 to the character select table 295 (Fig. 
13), at steps 122. 

It further determines, at step S1 23, the voice output so 
pattern of the agent's speaking from the communication 
type of the program that has been confirmed at S121, 
the agent's character that has been determined at S122 
and the current status (vehicle status, external factors, 
time, etc.), in reference to the voice pattern select table ss 
297 (Fig. 14). 

Thus, at step S124, agent processing unit 1 1 reads 
out, from the voice data memory 295, voice data neces- 



sary to synthesize the agent's speaking contents (see 
Fig. 3) of the selected communication program and 
tunes the synthesized speaking contents in accordance 
with the determined voice output pattern. Suppose that 
the program No. 00011 was selected at S121, in which 
the agent says "Good afternoon, I'm Lisa. May I go with 
you V (see Fig. 3). And, when the "respectfully-speak- 
ing" voice output pattern was selected at Si 23, agent 
processing unit 1 1 controls such that the agent (Lisa) 
says the above greetings relatively slowly. When the 
"cheerfully-speaking" voice output pattern was 
selected, the agent says the above greetings in a cheer- 
ful and a little bit higher tone. 

After such tuning control operation at S124, the 
communication program selected at S121 is now initi- 
ated, at step Si 25. The prescribed agent's action of 
selected communication program is shown on display 
27, while the prescribed agent' speaking thereof is out- 
putted through speaker 25 after subjected to proper tun- 
ing at S 124. 

In accordance with this embodiment, the agent is 
more personified to make human communication with 
the driver. Even when the same communication pro- 
gram is applied, the agent's speaking pattern changes 
in response to the current status, for example. When the 
vehicle is driving with the windows opened, the agent 
speaks loudly. When the vehicle is in a traffic jam on a 
rainy day, the agent is out of humor, whereas the agent 
speaks cheerfully during driving at high speed along a 
seaside road (that can be detected by navigation 
processing unit 1 0) on a fine day. 

In accordance with another embodiment of the 
present invention, the agent is personified by selectively 
controlling the agent's character depending upon the 
vehicle condition or other situation concerned. In this 
embodiment, even if the same communication program 
is selected, the agent's activity should be changed in 
response to change of situation. Moreover, the agent's 
activity is determined not only the situation but also the 
past record study. 

In this embodiment, processing unit 1 (Fig. 1) is 
connected to agent data memory 29b that stores agent 
program 290, program select table 291b, study item 
data 292, response data 293, image data 294, agent's 
character select table 296 and character-determining 
data 299, as shown in Fig. 17. Program 290 and data 
292-294 are substantially identical to those stored in 
agent data memory 29 of the first embodiment of Fig. 1 . 
Agent's character select table 296 is substantially iden- 
tical to that stored in agent data memory 29a of the sec- 
ond embodiment of Fig. 12. Accordingly, explanation 
thereof is not repeated herein. Character-determining 
data 299 is basic data for use in determining the agent's 
character in reference to character select table 296. 
Such basic data includes the vehicle type, the user's 
birthday, sex, character for every driver who may use 
the present system. 

Except for difference in contents of agent data 
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memories 29 and 29a, the device is almost similar to 
that shown in Fig. 1 and therefore no description of the 
other parts and elements of the device is given here to 
avoid tedious repetition. 

Agent data memory 29b in this embodiment com- s 
prises one or plural memory medium that can be 
selected from floppy disk, hard disk, CD-ROM, mag- 
neto-optical disk, magnetic tape, IC card, magneto-opti- 
cal card, etc. and drive unit for the memory medium 
used. It may be preferable that study item data 292, w 
response data 293 and character-determining data 299 
are stored in removable memory medium such as floppy 
disk and IC card and other data are stored in a hard 
disk. 

Program select table 291 b is similar to table 291 of is 
Fig. 2, but further includes the agent's character on the 
abscissa for some communication programs. This 
means that, in this embodiment, the agent's character 
that is determined in reference to character select table 
296 comprises one of factors for determining a commu- so 
nication program to be applied. 

In this embodiment, as described above, the agent 
activity of a specific communication program that is 
selected by table 291 may be represented in different 
manner depending on the agent's character determined 25 
by table 296. Suppose that a communication program 
designating the agent's activity of "falling down" is 
selected in response to the driver's emergency braking 
operation. Where the agent's character has been deter- 
mined as like a "crybaby", the agent falls down behind so 
her buttocks while snivelling or crying. A "beautiful sec- 
retary" do the substantially falls down but remains calm 
and says "Why?". An agent like a "high-school girl" 
screams in doing the action of "falling down". 

The flowchart of Fig. 15 also describes the main ss 
routine of the agent control operation of this embodi- 
ment. Description regarding the flowchart of Fig. 15 
should be referred to in understanding the agent control 
operation of this embodiment, in this embodiment, how- 
ever, the agent control operation that is carried out at 40 
S114 of the main routine of Fig. 15 is processed as 
shown by the flbvychkrt of Fig. 18. 

Now in referenpp to the flowchart of Fig. 18, agent 
processing unit; 111. (determines the agent's character by 
applying the alreadyspbtained current status (e.g., the 4s 
total number of j signition-on operation, the current 
weather, time, etc.) and basic data in character-deter- 
mining data memory 299 to character select table 296, 
at step S221. 

From the obtained current status data and the so 
agent's character determined at S221, agent process- 
ing unit 11 discriminates if there is any communication 
program to be initiated, in reference to program select 
table 291b, at step S222. If not (No at S222). the proce- 
dure is returned to the main routine of Fig. 15. Where ss 
there is a communication program to be initiated (Yes at 
S222), it confirms the program number thereof. It further 
reads out the driver's response record stored in 



response data 293 for the said program number to 
thereby determine if the communication program of the 
selected program number should be initiated or sus- 
pended, at step S223. 

rf the answer at S223 is No, agent processing unit 
1 1 operates to initiate the selected communication pro- 
gram so that the predetermined agent's activities are 
shown on display 27 and outputted through speaker 25, 
at step S224. This enables good communication 
between the driver and the agent which reflects not only 
the vehicle and driver's current status but also the result 
of study of the past status data and the driver's 
response records. When the driver makes any response 
to the agent's activities made at S224, such response 
data is obtained through microphone 26 or input device 
22, at step S225. 

Then, at S226, agent processing unit 1 1 discrimi- 
nates if the communication program made at S224 
involves any suggestion, proposal or question that 
awaits the driver's response or answer. For example, 
agent says "Do you want to hear any radio program ?" 
or "Do you feel hungry? May I find any restraint near 
here?". The latter program may be initiated in response 
to detection of a hungry sound through microphone 26. 

When the selected communication program is not a 
questioning program (No at S226), the procedure is 
advanced to step S228. When it is a questioning pro- 
gram (Yes at S226), at a step S227, agent processing 
unit 1 1 controls the agent's activities in accordance with 
the driver's response already obtained at S227. 
Regarding the example of operation at S227, the 
description of S1 9 should be referred to. 

Agent processing unit 1 1 accumulates data regard- 
ing the execution of the selected communication pro- 
gram to obtain the study data, at step S228. In 
connection with S228, the description of S20 should be 
referred to. Further, in this embodiment, when the 
response at S225 includes any data that influences 
determination of the agent's character, it is stored in 
character-determining data section 299 in agent data 
memory 29. After executing the operation at S228, the 
procedure is returned to the main routine of Fig. 15. 

Figs. 19A-19C show an example of the agent activ- 
ities that is controlled by agent processing unit 1 1 after 
the ignition switch is turned on. In this example, ignition 
sensor 401 detects that the ignition switch is turned on, 
current position sensor 21 detects the current vehicle 
position is other than the driver's home and rest posi- 
tions. These data are supplied to agent processing unit 
1 1 as the current status data at Si 1 3 of the main routine 
of Fig. 15. Regarding study item data 292 and response 
data 293, it has confirmed that this is the second igni- 
tion-on operation today, the last agent control operation 
was terminated at 10:05, on July 12, 1997, and the total 
number of ignition-on operation amounts to 30 times. 

From these data, agent processing unit 1 1 deter- 
mines the communication program No. 00004 should be 
executed for "beautiful secretary" and No. OOOxx for 
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"high-school girl", in reference to the program select 
table 291b, at S222 of the flowchart of Fig. 18. The 
agent activity to be outputted through display 27 and 
voice output device 25 is an "obeisance" that is a 
respectful bow, with saying "Hello, I'm Lisa. I will go with 5 
you again" in the communication program No. 0004 and 
"Hi, I'm Lisa. Let s go with me" in the communication 
program No. OOOxx. After confirming that the selected 
communication program should not be suspended (No 
at S223), agent processing unit 1 1 starts this communi- 10 
cation program (at S224), as shown in Fig. 19B. The 
agent's obeisance is represented by a series of station- 
ary images or animation. 

Since the selected program of No. 00004 or No. 
OOOxx is not a questioning program that awaits the 75 
driver's response (No at S226), the procedure is 
advanced to S228 where the study data is stored and 
renewed as a result of execution of the selected pro- 
gram. In this example, the number of today's ignition-on 
operation is changed from the previous value "2" to "3", so 
and the total number of ignition-on operation is changed 
from the previous value "30" to "31". Then, the proce- 
dure is returned to the main routine of Fig. 15. 

Fig. 20A represents a example of program select 
table 291b, and Fig. 20B represents the agent's activi- 25 
ties to be outputted in the communication program 
selected by table 291b of Fig, 20A. The communication 
program Nos. 02000-02003 shown in Fig. 20A is 
selected when the driver changes the shift lever from 
the parking position (P) to the driving position (D) while 30 
the side brake is still in operation and then pushes the 
accelerator pedal to a certain degree. The accelerator 
pedal operation, the side brake operation and the shift 
lever position are detected by sensors 403, 405 and 406 
(Fig. 8) in current status sensor 40 (Fig. 1), which agent 35 
processing unit 1 1 confirms at S113 of the main routine 
of Fig. 15. In the communication program 02000-02003, 
the agent calls the driver's attention to the current side 
brake position and immediate release thereof, which is 
accomplished by the agent's activities shown in Fig. 40 
20B. 

When the "Obliging beautiful secretary" has been 
selected as the ^gent's character (at S221 of Fig. 18), 
the communication program No. 02000 is selected in 
reference to program select table 291b of Fig. 20A (at 45 
S222). As shown in Fig. 20B, this communication pro- 
gram designates the agent activity such that the agent 
warns by saying "The side brake is still in operation. 
Release it, please", in a persuasive tone. Also, the 
agent's activity is shown on display 27 just like such that so 
her right hand is pulled back by the side brake. 

When the "high-school girl' has been selected, the 
communication program No. 02001 is applicable. In this 
case, the high-school girl on display takes the same 
reaction while saying "Oh nol The side brake is still ss 
working. Release it please" in a cheerful tone. When the 
"crybaby" has been selected, the communication pro- 
gram No. 02002 is applicable so that the crybaby takes 



the same reaction with a tearful face and says "Ouch I 
The side brake is still in operation. Release it, please" in 
a sadly and entreating tone. When the "placid" agent 
has been selected, the communication program No. 
02003 is applicable. In this case, the agent takes the 
same reaction and says "May I suggest you releasing 
the side brake" in a placid tone. 

In the foregoing embodiment, the agent's character 
changes depending upon the current status, the user- 
related data, the weather condition, time, etc. However, 
in a more simple manner, the agent's character may be 
selected from predetermined several characters 
depending on the driver's familiarity to the agent sys- 
tem. The driver's familiarity to the agent system is calcu- 
lated in a manner to be described hereinlater. The agent 
is "placid" to the driver who is much familiar to the agent 
system, whereas an "obliging" agent appears to the 
unfamiliar driver. The obliging agent provides sugges- 
tion, advice, help, explanation, etc. in a meddlesome 
manner rather than by the placid agent. 

In this modified embodiment, agent data memory 
29c stores familiarity data 300, as shown in Fig. 21. 
Agent's character select table 296 and character-deter- 
mining data 299 in agent data memory 29b are not 
present. Familiarity data 300 is stored for every driver 
who may use the agent system, and renewable by the 
past record study. In this modified embodiment, the 
agent character is "obliging" or "placid" so that only 
these two characters are included in the column of 
"Agent's Character" in program select table 291b (Fig. 
20 A). 

Fig. 22 shows an example of familiarity data 300. 
As shown, familiarity data 300 describes the program 
number, the flag, the last five time response period, the 
average response period, the shortest response period 
and the number of response cancellation for each com- 
munication program. The response period means a 
period of time from when the agent makes some ques- 
tion until when the driver makes some response thereto, 
or a period of time when the driver is expected to make 
some operation until the said operation is actually 
taken. The response periods for the latest five time 
responses are recorded in familiarity data 300. The 
average response period is the average of the five 
records. The shortest response period is the shortest 
one of the five records. When the driver cancels his or 
her response or operation while the selected communi- 
cation program is still working, it is accumulatively 
recorded as the number of response cancellation. The 
flag describes how many times this communication pro- 
gram has been selected. Familiarity data 300 is pre- 
pared for every communication program that requests 
the user's response, for every user of this agent system. 

The familiarity level is calculated from these data 
and also described in familiarity data 300. Of course, 
the familiarity level is zero when the flag is zero. Each 
time the communication program is selected, the famili- 
arity level is increased by 1 point. The average response 
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period becomes shorter than a predetermined period, 
the familiarity level is increased by 2 point. The shortest 
response period becomes lower than a predetermined 
period, it is increased by 1 .5 point. 

On the other hand, when the aiverage response s 
period becomes longer than a predetermined period, 
the familiarity level is decreased by 2 point. The shortest 
response period exceeds a predetermined period, it is 
decreased by 1 .5 point. It is decreased by 1 point, when 
the response cancellation is made by the driver. The 10 
respective points to be increased or decreased may be 
determined as desired. 

In this modified embodiment, agent processing unit 
1 1 determines the agent's character by confirming the 
familiarity level of the selected responsive program and 15 
of the current user, at S221 of Fig. 18. By way of exam- 
ple, when the familiarity level in familiarity data 300 (Fig. 
22) is below 40, the agent* character is determined as 
"obliging", whereas it is 70 or more the agent's charac- 
ter is determined as "placid". In accordance with the so 
agent's character thus determined at S221 , a specific 
communication program is selected at S222 followed by 
the succeeding steps. 

At S225, agent processing unit 11 confirms the 
driver's response to the agent's activity in the running 25 
program, which is made through microphone 26 or input 
device 22. It also confirms the response period from 
when the agent's activity is outputted through display 27 
and speaker 25 until when the driver's response thereto 
is obtained. From these, agent processing unit 11 30 
rewrite or renew the familiarity data 300. More particu- 
larly, in the column of the last five time response period, 
the oldest record is annulled, the remaining four records 
are moved to the older boxes one by one and the 
obtained response period is recorded as the latest one. 35 
Then, the average response period and the shortest 
response period are calculated and recorded. From the 
renewed records, the familiarity level is calculated by a 
predetermined formula and the newly obtained familiar- 
ity level is recorded in the familiarity data section 300 40 
which should be used in the next determination of the 
agent's character] I 

In accordance yyith still another embodiment of the 
present invention,; ia:plurality of agents may appear on 
display 27 and/bf Qi^utted through speaker 25 so that 45 
the user can easily Understand the communication, 
message, suggestion, etc. from the agents. 

In this embodiment, processing unit 1 (Fig. 1) is 
connected to agent data memory 29d that stores agent 
program 290, program select table 291 , study item data so 
292, response data 293, image data 294, help-agent 
program 301, help-agent program table 302 and famili- 
arity data 300, as shown in Fig. 23. Program 290, table 
291 and data 292-294 are substantially identical to 
those stored in agent data memory 29 of the first ss 
embodiment of Fig. 1 . Familiarity data is substantially 
identical to that stored in agent data memory 29c of the 
preceding embodiment of Fig. 21 and an example 



thereof is shown in Fig. 22. Accordingly, explanation 
thereof is not repeated herein. 

Help-agent program 301 comprises help-agent 
processing program and help-communication program 
especially for each responsive communication program 
stored in agent program section 290. Help-agent 
processing program is used to carry out help-agent 
operation which provides explanation or other suitable 
communication for easier understanding of the main- 
agent's activity, while some responsive communication 
program is working. Help-agent communication pro- 
gram describes the heip-agent's activities to be repre- 
sented through display 27 and speaker 25. 

Help-agent program 301 stores plural kinds of voice 
data for each main agent's voice of the main-communi- 
cation program, one of which may be selected by the 
driver's operation through input device 22, for example, 
so that the driver distinguishes the main-agent's and 
help-agent's voices both outputted through speaker 25. 
For example, the main-agent is female, the help-agent 
should be male; In a preferred example, after the main- 
agent's voice is determined, agent processing unit 1 
represents on display 27 a list of voice patterns for the 
help-agent which is distinguishable from the main- 
agent's voice, and the driver select one through input 
device 22. In the absence of the driver's instruction, 
agent processing unit 1 automatically determines the 
help-agent's voice that is distinguishable from the main- 
agent's voice. Likewise, the help-agent's appearance 
should be well distinguishable from the main-agent's 
appearance. 

Help-agent program table 302 represents corre- 
spondence between the (main-) agent program number 
and the help-agent program number, and the familiarity 
level that is critical to carry out the help-agent program. 
An example of the help-agent program table 302 is 
shown in Fig. 24, and some of the help-agent's activities 
in the help-agent programs in Fig. 24 are represented in 
Fig. 25. 

For example, when the agent's communication pro- 
gram No. 00123 (that suggests stop of the air-condi- 
tioner) is in operation, the help-agent program No. 
50010 is applicable provided that the driver's familiarity 
to this communication program is 50 or lower, in this 
help-agent program, the help-agent points his or her 
forefinger at the main agent (Lisa) on display 27 and 
says "Lisa stops the air-conditioner and opens the front 
windows half" through speaker 25. Even when the same 
communication program No. 00123 is working, when 
the driver's familiarity is higher than 50, the correspond- 
ing help-agent program No. 50010 is not operative and 
no help-agent appears. 

Fig. 26 is a flowchart of the main routine of the 
agent control operation in accordance with this embodi- 
ment. Steps S301-306 of this routine are substantially 
identical to S11-16 of the flowchart of Fig. 9, so that 
description of the fatter should be referred to as descrip- 
tion of the former. 
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After initiating the applicable communication pro- 
gram at S306. agent processing unit 1 1 checks and 
confirms the familiarity data 300 of the said communica- 
tion program in agent data memory 29, at step S307. At 
the same time, the familiarity flag is set to 1 . Then, the 5 
program number of the selected communication pro- 
gram and the confirmed familiarity data are applied to 
the help-agent program table 302 (Fig. 24) to discrimi- 
nate if there is any help-agent program to be initiated, at 
S308. If not (No at S308), the procedure is advanced to 10 
step S310 where the driver's response is obtained. 
Steps 310-316 are substantially identical to S 17-23 of 
the flowchart of Fig. 9 and, therefore, no detailed 
description is given herein. 

When some help-agent program to be initiated in 15 
the table 302 (Yes at S308), this is executed so that the 
prescribed help-agent's activity (action and utterance 
such as shown in Fig. 25) is outputted through display 
27 and speaker 25, at step S309. 

In accordance with this embodiment, the help- 20 
agent appears and supports the main-agent's activity 
when the driver is not so much familiar to the present 
agent system, whereby the driver can make more 
smooth communication with the agent and more suita- 
ble response to the agent's question or suggestion. 25 

Figs. 27A-27C illustrate an example of the agent 
control operation made by the flowchart of Fig, 26. This 
example is executed when the air-conditioner sensor 
418 detects "ON", the room temperature sensor 413 
detects the room temperature T1 and the open air tern- 30 
perature sensor 414 detects the open air temperature 
T2, as the current status obtained at S302. Operation at 
S303 reveals that the last two response to the commu- 
nication program No. 00123 are both C indicating the 
driver's acceptance so that the default temperature val- 35 
ues T3=2 and T4=24 have been read out from the study 
item data 292. Consequently/The relationship of (T1- 
T2)=£ T3 and T2 =£ T4 are both satisfied, as shown in Fig. 
27A. 

From these operation at S302 and S303, agent 40 
processing unit 11 finally selects the communication 
program No. Q0l!23 at S304. Then, after confirming that 
this program is rjot subjected to suspension at S305, it 
starts the communication program No. 00123 for repre- 
sentation of the (main-) agent's activity through display 45 
27 and speaker 25, at S306. 

Next, agent processing unit 1 1 confirms the driver's 
familiarity to the selected communication program at 
S307 and finds out an applicable help-agent program at 
S308. Since the driver's familiarity to the communication 50 
program No. 00123 is 25 (Fig. 27A) in this example, and 
the communication program No. 00123 has a corre- 
sponding help-agent program No. 50010 in the help- 
agent program table 302 (Fig. 24), this help-agent pro- 
gram No. 50010 is executed at S309. Accordingly, the ss 
help-agent H appears on display 27 in a predetermined 
appearance and speaks through speaker 25 in a prede- 
termined manner and tone. In this example, the main- 



agent (Lisa) E says "It becomes cool. I may suggest 
introducing the open air." and then the help-agent H 
says "Lisa stops the air-conditioner and opens the front 
windows half". The help-agent's activity provides sup- 
port and explanation of the main-agent's activity to the 
driver. Data necessary to display the help-agent activity 
is included in the image data 294. 

At S310 agent processing unit 11 obtains the 
driver's response. Since the communication program 
No. 00123 is a questioning or responsive program (Yes 
at S311), it controls such that the air-conditioner is 
stopped and the front windows are opened half in 
response to the driver's acceptance to the agent's sug- 
gestion (Fig. 27B), at S312. 

The driver's response is stored in response data 
293 for the communication program No. 00123, at 
S3 13. Since response data 293 stores the latest two 
response, the older response data is deleted and the 
newly obtained response data is stored together with 
one remaining data. As shown in Fig. 27C, when the 
driver's response is "neglect", application of this com- 
munication program is suspended one time, and the 
default temperature values T3 and T4 to be next applied 
are decreased by one. respectively. When the driver 
shows "refusal" application of this communication pro- 
gram is suspended five times, and the default tempera- 
ture values T3 and T4 to be next applied are decreased 
by one, respectively. In case of "acceptance", the same 
default values remain unchanged in the next applica- 
tion. 

Operation at S312 also includes re-calculation of 
the driver's familiarity to this communication program 
No. 00123. More particularly, the familiarity data 300 
(Fig. 22) is rewritten such that the oldest response 
period is deleted and the newly obtained response 
period (i.e., a period of time from when the agent 
appears on display 27 and until any driver's response is 
obtained). From the rewritten five response period data, 
new average response period, new shortest response 
period and new familiarity level are calculated and 
recorded in the respective memory areas. 

When the driver's familiarity to the communication 
program No. 00123 is grown to above 50, agent 
processing unit 1 1 discriminates there is no applicable 
help-agent program at S308, because the help-agent 
program table 302 teaches that the help-agent program 
No. 50010 corresponds to the communication program 
No. 00123 but should be executed only when the 
driver's familiarity is lower than 50. In this case, only the 
main-agent (Lisa) appears on display 27 while saying 
the prescribed suggestion to introduction of the open 
air. in accordance with the communication program No. 
00123. Then, agent processing unit 11 obtains the 
driver's response at S310, make suitable control in con- 
formity to the driver's response at S31 1 and rewrites the 
response data 293 and the familiarity data 300, as in the 
above-described manner. 

The above-described embodiment may be modified 
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as follows. In this modification, a plurality of sub-agents 
may appear depending on the driver's trend. A typical 
example of the sub-agent is a short-cut agent who 
appears, when the driver is much familiar to a specific 
communication program, for short-cut or skip of a part 5 
of the agent's lengthy activity of the same communica- 
tion program. 

In this modification, agent data memory 29d of Fig. 
23 is replaced with agent data memory 29e of Fig. 28. 
In agent data memory 29e are stored agent program 10 
290, program select table 291, study item data 292, 
response data 293, image data 294, sub-agent program 
303, sub-agent program table 304, trend data 309 and 
frequent program data 305, as shown in Fig. 28. Since 
the former five memory contents have been described is 
in connection with the preceding embodiments, the last 
two will be described in detail. 

Sub-agent program 303 comprises sub-agent 
processing program and sub-agent communication pro- 
gram especially for each communication program so 
stored in agent program section 290. Sub-agent 
processing program is used to carry out sub-agent 
operation wherein a sub-agent appears together with a 
main agent depending on the driver's trend during oper- 
ation of some communication program. Sub-agent com- 25 
munication program describes the sub-agent's activities 
to be represented through display 27 and speaker 25. 
Sub-agent processing program at least includes a pro- 
gram for achieving the short-cut agent operation that 
short-cuts or skips the 1 main-agent's lengthy activity 30 
when the driver is so familiar thereto. 

In another example, a specific trend value is 
obtained from the driver's birthday and the traffic jam 
condition so that the today's fortune is noticed to the 
driver when the vehicle is in a heavy traffic jam. 35 

In still another example, data regarding the driver's 
favorite professional baseball team and the radio pro- 
gram broadcasting the said team's games are stored as 
the driver's trend data 309. The sub-agent program is 
executed at the time when the radio program is to start, 40 
so that the sub-agent announces the radio program and 
the opponent team £nd ask if the radio receiver should 
be turned on. The r^dio program data may be inputted 
by the driver or autqitiatically received from communica- 
tion control unit 24, - 45 

Another driver'^ trend data may be determined from 
the current vehicle position and the places of the driver's 
interest so that the sub-agent draws the driver's atten- 
tion to some recorded place near the current vehicle 
position. Still another driver's trend data may be deter- so 
mined from the driver's driving career and the current 
vehicle position so that the sub-agent announces "Be 
careful, you are driving on a winding road", for example. 

Most typical example of the sub-agent program will 
be the short-cut agent program to be described in more 55 
detail. 

After executing the sub-agent program at S408, 
agent processing unit acquires the driver's response, at 



S409. As described before, if the driver's trend data is 
lower than the predetermined value (No at S407), no 
sub-agent program is executed and the driver's 
response is obtained at S409. The following steps 
S410-S415 are substantially identical to S18-S23 of Fig. 
9. 

The activity in the sub-agent program will now be 
described in detail. In the following description, the sub- 
agent is the short-cut agent. As shown in Fig. 33, the 
communication program for menu selection comprises 
multi-stage operation. The first-stage operation is mode 
select operation. The second-stage operation includes 
goal select mode and system select mode, one of which 
may be selected by the driver's operation through input 
device 22 when the first stage operation is working. 
When the goal select mode is selected, then the driver 
may select and enter one of third-stage operation, that 
is, how to select the goal, from its category (highway 
lamps, railroad stations, hotels, restraints, golf courses, 
etc.), from its phone number or from its spelling. When 
the system select mode is selected, then the driver may 
select one of third-stage operation for setting display, 
voice or communication in the agent system. In the nor- 
mal operation, operation begins with at the first stage, 
proceeds through the second stage and terminates at 
the third stage. 

Fig. 34 is similar to Fig. 33 but different in that there 
is a short-cut program shown by dotted lines. In this 
embodiment, after selecting the first-stage operation for 
mode selection, the driver may skip over the second- 
stage operation to select one of the third-stage opera- 
tion by using the short-cut program. The short-cut pro- 
gram is applicable when the driver is so familiar to the 
communication program, or when the driver's familiarity 
to the communication program is 75 or more in this - 
example. 

When the driver so operates to start the navigation 
system, agent processing unit 1 1 first performs the first- 
stage operation, that is the mode select operation. The 
communication program No. 00540 now becomes oper- 
ative (at S406) so that the main agent appears on dis- 
play 27 and says "May I soon start navigation program 
or provide you system select program?". Since this 
communication program has no corresponding short- 
cut program (No at S407), it awaits the driver's 
response at S409. When the driver requests navigation 
program In reply to the agent's activity of this program 
No. 00540 through input device 22 or microphone 26, 
this is confirmed at S409. This program No. 00540 is not 
subjected to the short-cut control (No at S410), and 
agent processing unit 11 carries out the following oper- 
ation at S41 1-415, then the procedure is returned to 
S402. 

The communication program No. 00550 for select- 
ing the goal input mode is now executed at S406. This 
means that the operation proceeds to the second-stage 
operation of Fig. 33. In this program No. 00550, the 
main agent appears on display 27 and says "Select your 
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goal input mode. From its category, from its phone 
number or from its spelling?", as shown in Fig. 36. 
Agent processing unit 11 then compares the driver's 
familiarity (75) with the predetermined critical value (70 
in Fig. 29) at S407. Since the former is larger than the 5 
latter in this example (Yes at S407), in reference to the 
short-cut program table 304 in Fig. 29, agent processing 
unit 11 executes the short-cut operation in accordance 
with one of the short-cut programs selected in accord- 
ance with the frequent program of the last five program w 
selection (at S408). 

Fig. 35 is a flowchart of the short-cut agent control 
operation at S408. As shown, agent processing unit 1 1 
obtains which is the most frequent program, at S481 
from the frequent program data 305 in agent data mem- is 
ory 29. Since the most frequent program is No. 00580 in 
the example of Fig. 31 , which means that the driver nor- 
mally inputs the goal from its category in response to 
the agent's activity of the mode select program No. 
00540, agent processing unit obtains the short-cut pro- 20 
gram No. 80010 in reference to Fig. 30, at S482, and 
then executed the same at S483. By executing the 
short-cut program No. 80010, the short-cut agent S 
appears on display 27 beside and behind the main 
agent E, and says "You can readily enter the categorical 25 
input mode." following the main agent's speaking, as 
shown in Fig. 36. The short-cut agent then awaits the 
driver's response. 

When the driver responsively says "Restraints", this 
is confirmed by the short-cut agent. In this case, agent 30 
processing unit 11 compulsory terminates the main 
agent communication program No. 00550 without reev- 
ing any response from the driver, and execute another 
communication program No. 00600 (at S404). The com- 
munication program No. 00660 is prepared as the 35 
fourth-stage operation after selecting the categorical 
input mode operation 00580 as one of the third-stage 
operation (Figs. 33 and 34). Thereafter, the current 
vehicle position is detected and the restraint map near 
the detected current vehicle position is shown on dis- 40 
play 27 for selective input by the driver. 

The driver niay say "Category" in response to the 
main- and sub-agent's activities at S483. In this case, 
the procedure! is advanced to the categorical input 
mode operation bf the program No. 00560 which is exe- 45 
cuted at S404 The list of selectable categories is repre- 
sented on display 27 and the main agent says "Select 
category of the goal". When the driver selects a specific 
input mode, the operation proceeds in the above- 
described manner. so 

On the other hand, if the agent's familiarity is 
smaller than 70 (No at S407), agent processing unit 1 1 
does not execute the short-cut program. In this case, 
only the main agent appears and says in accordance 
with the main agent communication program that has ss 
been determined at S404, as shown in Fig. 37. 

In this embodiment, the short-cut agent appears 
together with the main agent to suggest the short-cut 



operation to the driver familiar to the agent system. 
When the driver forgets the short-cut operation, he or 
she may follow the main agent's activity to successively 
carry out the multi-stage operation stage by stage. 

The sub-agent may act independently of the main 
agent's activity. For example, the main agent acts as a 
guide or navigator whereas the sub-agent notices the 
current vehicle condition to the driver, and the vice 
verso. 

In accordance with yet another embodiment of the 
present invention, one or more of personified agent 
appears on display 27 together with a predetermined 
background so that the user can easily understand the 
communication, message, suggestion, etc. from the 
agents. The agent's activity and the background are 
determined depending on the current status of the vehi- 
cle, the driver and passengers, etc. as well as the 
driver's response. 

In this embodiment, processing unit 1 (Fig. 1) is 
connected to agent data memory 29f that stores agent 
program 290, program select table 291 , study item data 
292f, response data 293, image data 294, background 
select table 306 and response recognition data 307, as 
shown in Fig. 38. Program 290, table 291 and data 293- 
294 are substantially identical to those stored in agent 
data memory 29 of the first embodiment of Fig. 1. 
Accordingly, explanation thereof is not repeated herein. 

The study item data 292f in agent data memory 29f 
is shown by example in Fig. 5 which is slightly different 
from the study item data 292 (Fig. 5) in agent item data 
292 (Fig. 1). That is, the study item data 292f also 
includes audio instruments equipped in the vehicle and 
their operating condition. Audio instruments may com- 
prise a radio receiver, a CD player, a MD player, a cas- 
sette tape player, a TV set, etc. The audio instrument 
operating condition describes when and where any 
audio instrument is turned on. If the radio or TV is 
turned on, the selected channel is also recorded. The 
audio instrument operating condition have five time 
records. The remaining items in the study item data 
292f have been described in connection with the first 
embodiment. 

In this embodiment, image data 294 include data 
for representing the background on display 27, as well 
as data for the agent's appearance and activity. The 
photo image taken by camera 28 is analyzed by image 
processing unit 13 to be used to represent the agent's 
activity or the background. The background image data 
294 may be photo, animated or any other image data 
representing sunrise, sunset, starlight night, snowy 
mountain, autumnal tints, golf course, plateau, etc. 
Among these, sunrise, sunset and starlight night back- 
grounds may represent the current time, snowy moun- 
tain and autumnal tints may represent the current 
season and golf course and plateau background may 
represent the goal inputted by the driver in the naviga- 
tion system. The background may also be the agent's 
belongings, such as nameboards showing the driver's 
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selectable response or answer "YES" and "NO", which 
is applicable when some questioning communication 
program is in operation. When the driver is listening to 
the music through CD or radio, a background like a 
musical note may be shown on display 27. 5 

An example of the background select table 306 is 
shown in Fig. 40. The items on the ordinate are various 
background, among which one or more of specific back- 
ground is selected by the items on the abscissa. The 
items on the abscissa include, in this example, the cur- io 
rent time, the current season, the instruments now in 
operation, the voice recognition status, the most fre- 
quent recognized word, the goal, etc. These items may 
be detected by various sensors included in status sen- 
sor 40, alone or in combination with the contents in the 75 
study item data 292 and/or response data 293. In this 
table, the item symbolized by "O" * s a necessary con- 
dition for selecting the background on the ordinate. The 
item with no symbol should not be satisfied to select the 
background on the ordinate. 20 

Figs. 41 A-41 C show examples of the background to 
be selected by the background select table 306, which 
is represented together with the agent's activity on dis- 
play 27. In Fig. 41 A, the background with a red frame 
(see Fig. 40) is selected when the driver's voice is not 25 
recognizable by the agent. The agent in Fig. 41 A is sit- 
ting down, which represents that the vehicle is now at a 
standstill. The background is shown in Fig. 41 B with a 
yellow frame (see Fig. 40) which indicates that the 
agent awaits the driver's answer by saying "YES" or 30 
"NO". In this background condition, the driver's voice 
other than "YES" and "NO" can not be recognized by 
the agent. The agent in Fig. 41 B carries two name- 
boards of "YES" and "NO" while saying "Please answer 
by saying "YES" and "NO" to my question", for example. 35 
This agent's activity is outputted when some question- 
ing communication program that request the driver's 
answer. The agent's belongings such as the above 
nameboards should be a kind of the background, as 
noted from Fig. 40. 40 

When the communication program for goal input is 
running and ltrie driver selects to input the goal by its 
category and then fcjy its distinct, the agent appears on 
display 27 with ajinkmeboard of "DISTRICT* for urging 
the driver to input afepecif ic name of the distinct of the 45 
goal, as shown in Fig. 41 C. The driver's input may be by 
voice through microphone 26 or by operation of input 
device 22. At the same time, a pertinent map is shown 
on display 27, and the frame of display 27 is colored 
green for indicating that any driver's voice is recogniza- so 
ble. In this example, the "DISTINCT" nameboard. the 
background map and the green frame are represented 
as the backgrounds that have been selected by the 
table 306 (Fig. 40). Fig. 8D merely shows another 
example of the background with a green frame indicat- ss 
ing voice recognizable condition, as above described. 
The agent is standing upright in Figs. 41B-41C which 
indicates that the vehicle is running. 



The response recognition data 307 represents a 
result of the agent' study of the response from the 
driver. The response recognition data 307 is prepared 
for each driver and rewritten as the driver makes some 
fresh response. 

An example of the response recognition data 307 is 
shown in Fig. 42. As shown, the response recognition 
data 370 includes the response recognition results, dis- 
tribution of the recognized response patterns and the 
most recognizable response pattern, respectively for 
affirmative and negative responses. The affirmative 
response may be "Yes", "OK", "Well", "Certainly", 
"Sure", etc. and the negative response may be "No", 
"You're Kidding". "Not at all", etc. The response recogni- 
tion results records in respective answer columns "O " 
or "X" symbols indicating if such driver's answer could 
be correctly recognized in the agent system, for past 10 
times recognition. The driver's answer inputted through 
microphone 26 is processed by voice processing unit 
14, and a signal after voice processing is supplied to 
agent processing unit 11 for executing some control 
operation. If the driver's answer has been recognized in 
error, the driver's should make any surprising response 
to the executed control operation. From this, it could be 
judged that the last time voice was recognized in error. 

The distribution percentage is determined by divid- 
ing the total number of voice acquisition by the number 
of correct voice recognition, followed by being multiplied 
by 1 00. The driver's answer having the greatest distribu- 
tion percentage is recorded as the most recognizable 
answer. Thus, two nameboards describing the most rec- 
ognizable affirmative and negative answers in their 
response recognition data 307 (Fig. 42) are shown as 
the background of display 27, in case of Fig. 41 B, for 
example. 

The main routine of the agent operation of this 
embodiment is the same as shown in Fig. 15 and, there- 
fore, description regarding the flowchart of Fig. 15 
should be referred to in understanding the agent control 
operation of this embodiment. In this embodiment, how- 
ever, the agent control operation that is carried out at 
S114 of the main routine of Fig. 15 is processed as 
shown by the flowchart of Fig. 43. In he flowchart of Fig. 
43, steps S501 -502 are substantially identical to S14-1 5 
of the flowchart of Fig. 9, so that description of the latter 
should be referred to as description of the former. 

After confirming that the applicable communication 
program should be executed in this operation (No at 
S502), agent processing unit 11 discriminates if the 
applicable communication program is an instrument 
control program, at step S503. The instrument control 
program means a program to be executed to control 
operation of the vehicle or any instrument mounted to or 
equipped in the vehicle. If this is the case (Yes at S503), 
the said program is executed to carry out necessary 
operation under control, at S509. The result of the con- 
trol operation is stored in the study item data 292. 

When the applicable communication program is not 
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an instrument control program (No at S503), at step 

5504, agent processing unit determines a display back- 
ground, in reference to the background select table 306 
(Fig. 40), from the current status (the current time, sea- 
son, instruments in operation, status of driver's voice 5 
recognition, goal inputted in the navigation system, etc.) 
and the response recognition data 307 in agent data 
memory 29. The applicable communication is executed 

by displaying the selected background, together with 
the prescribed agent's activity, on display 27, at S505. w 
The agent may also talk to the driver through speaker 
25 in accordance with the contents of the applicable 
communication program. Thus, the agent's activity 
(action and voice) such as shown in Figs. 41A-41D is 
shown on display 27 with a suitable background that is 75 
pertinent to the agent's activity. 

When this communication program is not a ques- 
tioning or responsive one requiring the driver's any 
response (No at S506), the procedure is returned to the 
main routine. When this is the case (Yes at S506), agent 20 
processing unit 1 1 acquires the driver's response or 
answer through microphone 26 and voice processing 
unit 14 or through the driver's operation of input device 
22, at step S507. The acquired response is incorpo- 
rated into response data 293 at step S508, then the pro- 25 
cedure is returned to the main routine. The data 
accumulation or the past record study at S508 has been 
described in connection with the preceding embodi- 
ments. In this embodiment, when the driver's response 
acquired at S508 is any reaction to the agent's activity 30 
that is executed in response to the last time driver's 
affirmative or negative answer in reply to the agent's 
questioning activity, this reaction is stored in the 
response recognition data 298 so that the distribution 
and the most recognizable answer are rewritten when 35 
necessary, as having been described. 

Figs. 44A-44C show examples of the agent's activ- 
ities shown on display 27, with the pertinent back- 
ground, in the communication program executed at 

5505. Fig. 44A represents the current status data 40 
acquired at S1 1 3 of the main routine of Fig. 15 and con- 
tents of the* stud jr item data 292 and the response data 
293 in agent dat^ memory 29. The acquired current sta- 
tus data includes the current time (17:00), the current 
vehicle position' (Tokyo) that is detected by the vehicle 45 
position sensor 21:. The study item data 292 and the 
response data 293 storing the past record study repre- 
sents in this example that the driver has most frequently 
been listening to the FM radio program "J-WAVE" dur- 
ing 17:00-18:00. so 

From these data in Fig. 44A, agent processing unit 
1 1 selects a communication program in reference to the 
program select table 291 at S501 . In this case, the com- 
munication program No. 00501 in which the agent sug- 
gests the driver listening the radio program is selected ss 
at S501 . This program is not subjected to suspension or 
skip, which is confirmed at S502. Then, it is confirmed 
that this program is not an instrument control program at 



S503. Agent processing unit 11 then determines a 
background to be displayed together with the agent's 
activity in the selected communication program No. 
00501, at S504. The background selection is made in 
reference to the table 306 of Fig. 40. For instance, since 
the current time is between 15:00-18:00, a sunset back- 
ground is selected. Further, this program No. 00501 
involves the agent's speaking, and the driver's voice 
spoken at the same with the agent's speaking would 
hardly be recognized. Accordingly, when executing the 
program involving the agent's speaking, display 27 has 
a red frame indicating the voice unrecognizable situa- 
tion now. Thus, a red frame background is also selected 
at S504. 

After one or more background has been selected in 
this way, the communication program No. 00501 is exe- 
cuted at S505 so that the agent's activity and the 
selected background are shown on display 27, as 
shown in Fig. 44B. More particularly, the agent appears 
on display 27 with a sunset background and says "You 
may listen to the radio program?". This communication 
program does not need the driver's response (No at 
S506) so that the procedure is returned to the main rou- 
tine of Fig. 15. 

In the next application of the main routine of Fig. 1 5, 
the driver's answer awaiting condition created by the 
program No. 00501 is newly confirmed at S1 13 of Fig. 
15 as one of the current status. As a consequence, 
another communication program is selected at S501 for 
obtaining the driver s answer in response to the agent' 
speaking (Fig. 44B). Through S502 (No) and S503 (No), 
at S504, agent processing unit 1 1 determines the back- 
ground to be applied in executing this program, in refer- 
ence to the background select table 306 (Fig. 40). In this 
case, the current time is within the period 15-18, the 
program to be executed requires recognition of the 
driver's affirmative or negative answer, and the most 
recognizable answers (in the response recognition data 
307 of Fig. 42) are "Yes" and "No". Accordingly, a sunset 
background, two nameboards of "YES" and "NO" as the 
agent's belongings and a yellow frame (indicating that 
only the driver's affirmative and negative answers can 
be recognized) are selected as backgrounds, which are 
shown on display 27, together with a silent agent, as 
shown in Fig. 44C, when this program is actually exe- 
cuted at S505. Since this program is a questioning pro- 
gram (Yes at S506), the driver's answer is obtained at 
S507. In this example, it is supposed that the driver says 
*Yes" in response to the agent's activity of Fig. 44C. This 
response data 293 is renewed accordingly at S508, and 
the procedure is returned to the main routine of Fig; 15. 

In the next application, the fact that the driver made 
an affirmative answer in response to the agent's activity 
in the previously executed communication program is 
confirmed at S113 as one of the current status. As a 
consequence, still another communication program is 
selected at S501 for turning the radio receiver on and 
selecting the radio channel (Fig. 44B). This program 
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should not be suspended (No at S502). Since this is an 
instrument control program (Yes at S503), the radio 
receiver is controlled accordingly at S509. In execution 
of this program, the radio receiver is automatically tuned 
to the most frequently selected channel in the current $ 
area and at the current time in accordance with the 
study item data 292. Thereafter, data regarding the 
radio in the study item data 292f (Fig. 39) is rewritten to 
include data in the latest operation. 

Unless the driver makes any reaction to ask inter- 10 
ruption of operation of the radio receiver by saying "No" 
or "Stop", for example, agent processing unit 1 1 dis- 
criminates that the driver's answer ("Yes") has been cor- 
rectly recognized. Thus, the response recognition data 
307 (Fig. 42) should be renewed by incorporating this is 
answer at S508, and the procedure is returned to the 
main routine of Fig. 15. 

In the next application, the operating condition of 
the radio is confirmed at S1 13 as one of the current sta- 
tus. Still another communication program may be 20 
applied to represent the current vehicle status (the sta- 
tus of the radio, in this case) at S501 . Through S502 and 

5503. one or more pertinent background is selected at 

5504. In this case, the present time is 17:00, the radio is 

in operation and any voice is recognizable, so that a 25 
sunset background with a green frame (indicating any 
voice recognizable condition) is shown on display 27 
together with a silent agent carrying a radio/cassette, at 

5505. Since this program is not a questioning program 
(No at S506), the procedure is soon returned to the $0 
main routine of Fig. 15. 

In accordance with the above-described embodi- 
ment, not only the agent but also the background perti- 
nent to the current status and/or the agent's activity are 
represented on display 27. This enables more informa- 35 
tion to be displayed at the same time and more mes- 
sage to be given to the driver. The agent and the 
background may represent different information or mes- 
sage. The background may represent where and how 
the vehicle is driving. For example, a tree shown as a 40 
background may be bent to the wind (to the left). In 
anther example, iwHere the vehicle is running on an 
uphill or downhill,' sych a road condition may be repre- 
sented as a background. 

In accordance, with yet another embodiment of the 45 
present invention, even when the same communication 
program is executed, the agents appearance and/or 
activity will be changed depending on the current status 
of the vehicle, the driver and the passengers, etc. and 
as a result of the study of the current status and the so 
driver's response, so that the user can make more 
friendly communication with the agent. 

In this embodiment, processing unit 1 (Fig. 1) is 
connected to agent data memory 29g that stores agent 
program 290. program select table 291 , study item data 55 
292f, response data 293, image data 294, appearance 
select table 308 and familiarity data 300, as shown in 
Fig. 45. Program 290, table 291 and data 292-294 are 



substantially identical to those stored in agent data 
memory 29 of the first embodiment of Fig. 1 . Accord- 
ingly, explanation thereof is not repeated herein. The 
familiarity data 300 has also been described in detail in 
reference to Fig. 22. The image data 294 in this embod- 
iment stores a great number of appearance such as 
looks, features, wears, belongings, situation, posture, 
altitude, etc. to be applied to the agent appearing on dis- 
play 27. When the driver selects his or her favorite char- 
acter through input device 22 or any other input means, 
the selected agent appears on display 27 with appear- 
ance selected in reference to the appearance select 
table 308. 

Fig. 46 is an example of the appearance select 
table 308. When the vehicle current status such as 
described in the left-hand columns of the table 308 is 
detected by a corresponding sensor of the current sta- 
tus sensor 40 (Fig. 1), a specific appearance of the 
agent at the neighborhood center column should be 
selected. The agent's appearance may also be deter- 
mined by the past record study stored in the study item 
data 292 and/or the response data 293, in which case 
such data contents should be described in the left-hand 
columns. Each appearance selectable in the table 308 
has the order of priority. When a plurality of appear- 
ances are applicable which can not be executed at the 
same time, one of them having the earlier order of prior- 
ity is selected. For example, when the wiper is working 
on Christmas Eve, appearances of "wearing a raincoat", 
"wearing a sweater" and "dressed as Santa Claus" are 
applicable as the agent' wear, in which case the appear- 
ance of "wearing a raincoat" is selected according to pri- 
ority so that the agent wearing a raincoat appears on 
display 27. If the wiper is turned off, the appearance of 
the next priority is selected so that the agent appears 
under the disguise of Santa Clause. 

The contents of the appearance select table 308 
shown in Fig. 46 is only example and of course not imi- 
tative. For example, it may be possible that the type of 
the vehicle is detected when the ignition switch is turned 
on, and the agent wears a formal suit when the vehicle 
is a high-grade sedan whereas the agent wears a T-shirt 
when the vehicle is a RV (recreation vehicle) or SUV 
(sport utility vehicle). In another example, the agent's 
height changes with the fuel remainder. The agent may 
jump up when VSC or ABS becomes operative. The 
agent may be inclined while excessive acceleration, 
deceleration or lateral gravity is applied to the vehicle. 
The agent may appear with a card indicating television 
or radio channels when the driver turns the television or 
radio on. The agent may become tall or fat as the driver 
becomes more familiar to the navigation system. The 
agent's wear or other appearance may be changed 
depending on the goal. The agent may perspire when 
the room temperature is higher than the open air tem- 
perature by a predetermined difference. The agent' 
shoes may be changed depending on the road condi- 
tion, more particularly the agent wears a pair of moun- 
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taineering or trekking boots white the vehicle is running 
on an unpaved road. The environmental data at the left- 
hand column may include the traffic jam, rainfall, snow- 
fall, sunshine, the number of passengers, etc. The 
user's data may include the driver's occupation. For 5 
example, the agent wears a white overall when the 
driver is a doctor. The user's data may also include the 
driver's or passenger's birthday, payday blood type, 
annual income, family members, nationality, native lan- 
guage, religion, favorite music, favorite sport, etc. Data 10 
regarding date and time may include anniversary, holi- 
days, lunch time, coffee brake time, dinner time, etc. 
which is known to the driver by a pertinent agent's 
appearance. 

The main routine of the agent operation of this 75 
embodiment is the same as shown in Fig. 1 5 and, there- 
fore, description regarding the flowchart of Fig. 15 
should be referred to in understanding the agent control 
operation of this embodiment. In this embodiment, how- 
ever, the agent control operation that is carried out at 20 
S114 of the main routine of Fig. 15 is processed as 
shown by the flowchart of Fig. 47. In the flowchart of 
Fig. 47, steps S60 1-602 are substantially identical to 
S14-15 of the flowchart of Fig. 9, so that description of 
the latter steps should be referred to in understanding 25 
the S601-602 of this flowchart. 

After confirming that the applicable communication 
program should be executed in this operation (No at 
S602), agent processing unit 11 discriminates if the 
applicable communication program is an instrument 30 
control program, at step S603. The instrument control 
program means a program to be executed to control 
operation of the vehicle or any instrument mounted to or 
equipped in the vehicle. If this is the case (Yes at S603), 
the said program is executed to carry out necessary 35 
operation under control, at S609. The result of the con- 
trot operation is stored in the study item data 292. 

When the applicable communication program is not 
an instrument control program (No at S603), at step 

5604, agent processing unit determines an agent's 40 
appearance, in reference to the appearance select table 
308 (Fig. 46), jfroh the current status (vehicle data, nav- 
igation data, environmental data, user's data, current 
date and trnjie^ : result of driver's voice recognition, 
driver's familiarity to the agent system etc.). The appli- 45 
cable communication is executed by displaying the 
agent with the selected appearance, on display 27, at 

5605. The agent may also talk to the driver through 
speaker 25 in accordance with the contents of the appli- 
calbe communication program. Thus, the agent's activ- so 
ity (action and voice) such as shown in Figs. 48B, 49B 
and 50 is shown on display 27 with a suitable appear- 
ance that is pertinent to the agent's activity or the cur- 
rent status. 

When this communication program is not a ques- ss 
tioning or responsive one requiring the driver's any 
response (No at S606), the procedure is returned to the 
main routine. When this is the case (Yes at S606), agent 
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processing unit 11 acquires the driver's response or 
answer through microphone 26 and voice processing 
unit 14 or through the driver's operation of input device 
22, at step S607. The acquired response is incorpo- 
rated into response data 293 at step S608, then the pro- 
cedure is returned to the main routine. In this 
embodiment, especially, agent processing unit 11 
acquires at S607 the driver's response to the agent's 
activity in the executing communication program and a 
time elapse from when the agent' activity is outputted 
until when the driver's response is confirmed. The data 
accumulation or the past record study at S608 has been 
described in connection with the preceding embodi- 
ments. In this embodiment, when the executed commu- 
nication program is a questioning or responsive 
program that needs the driver's response, the response 
obtained at 608 is incorporated into the response data 
293. When the response data 293 already has full 
records, the oldest record is deleted and replaced by 
the newly obtained record. Further, the familiarity data 
300 (Fig. 22) is renewed to a new value, as in the man- 
ner having been described hereinbefore in connection 
with the preceding embodiments. 

The agent's appearance and activity in this embod- 
iment will be described hereunder in reference to Figs. 
48A-50 by way of example. Ftgs. 48A-48C illustrates an 
example of the agent control operation executed after 
the ignition switch is turned on. As shown in Fig. 48A, 
agent processing unit 1 1 acquires, as the current status, 
a time elapse T1 from the ignition-on from a detection 
signal from a timer (one of sensors 40). It has also been 
confirmed that current vehicle speed is 35kg/h, the 
driver's voice is recognizable, the wiper is working, the 
current season is summer, the current driver's familiarity 
is 27, etc. Further, agent processing unit 1 1 make refer- 
ence to the response data 293 to confirm that the driver 
showed "acceptance" to the agent's activity in the latest 
two application of the program No. 00123. The current 
default value TO stored in the study item data 292 is 1 20 
minutes. Then, it has also been confirmed that the 
elapsed time T1 ^ TO. 

From these data obtained and confirmed, agent 
processing unit 1 1 finds the applicable communication 
program No. 00123 (Yes at S601) and confirms that 
application of this program should not be suspended at 
this time (No at S602). It also confirms that this program 
is not an instrument control program (No at S603). 

Then, agent processing unit 1 1 determines the 
agent's appearance from the current status of Fig. 48 A, 
at S604. The appearance select table 308 of Fig. 46 
teaches that, in these current status, the appearances 
of "hair falls down naturally", "full face", "wear a swim- 
suit", and "tall" should be selected. Accordingly, when 
executing the program No. 00123 at S605, a tall agent 
wearing a swimsuit (or a sportswear) stands naturally 
with her full fact, on display 27. At the same time, she 
talks to the driver "'Will you have a rest?" through 
speaker 25. 
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Since this program is a questioning program (Yes at 
606), the driver's response is obtained at S607, as 
shown in Fig. 48C. The driver's response "neglect", 
"refusal" or "acceptance", together with date and time of 
the response, is stored in the response data 293 of the $ 
program No. 00123 at S608. The oldest response data 
is replaced with the newly acquired response data. 
When the driver shows "neglect", the default value 
increased by 10 minutes is applied in the next but one 
application of this program. In case of "refusal", the 10 
default value increased by 20 minutes is applied in the 
next but five application of this program. In case of 
"acceptance", the same default value is applied in the 
next application without suspension. These are the 
result of the past record study to be incorporated in the 15 
response data 293. Then, the procedure is returned to 
the main routine of Fig. 15. 

Fig. 49A illustrates the different current status, in 
which case the agent's appearance such as shown in 
Fig. 49B is selected in reference to the table 308 of Fig. so 
46. In this case, the appearance of "hair falls down nat- 
urally", "face in profile", "wear a sweater", and "short" 
should be selected. Accordingly, when executing the 
program No. 00123 at S605, a short agent wearing a 
sweater faces in profile, on display 27, and says "Will ss 
you have a rest?" through speaker 25. Operation at 
other steps will be substantially the same as in the 
examples of Figs. 49A-49C. 

In the above-described embodiment, a plurality of 
selectable agent's appearance has respective image 30 
data stored in the image data memory section 294. 
However, it may be possible that one or several image 
data is prepared for one communication program and 
changed by image processing unit 13 in accordance 
with the current status. Change of the agent's appear- 
ance may include change or the agent itself. For exam- 
ple, the agent is a rabbit when the vehicle is running at 
a high speed whereas the agent is a turtle during low- 
speed drive. When the current status is that the driver's 
voice is recognizable, the agent may appear on display 
27 carrying a nameboard of "PLEASE" or two name- 
boards of "Y£S" andf "NO" (Fig. 50) for urging the driver 
to make a suitable response to the agent's activity in the 
last communication jprogram. 

Claims ; 

1 . A device mounted in a vehicle, comprising first sen- 
sor means for detecting predetermined current sta- 
tus; first memory means for storing study data 
renewable in response to entry of the latest current 
status detected by said first sensor means; at least 
one personified agent communi eatable with user in 
the vehicle; second memory means for storing a 
plurality of communication programs; communica- 
tion program select means for determining one of 
said communication programs to be executed in 
accordance with the current status detected by said 



first sensor means and/or the study data currently 
stored in said first memory means; agent control 
means for controlling activity of said agent in 
accordance with said selected communication pro- 
gram; agent output means for outputting said 
agent's activity to said user in the vehicle deter- 
mined by said agent control means. 

2. A device according to daim 1 wherein said first sen- 
sor means detects at least one vehicle condition 
selected from the group consisting of a current 
vehicle position, a current vehicle speed, a current 
shift position, winker operation, wiper operation, a 
coolant temperature, an engine oil temperature, an 
engine oil pressure, a passenger room tempera- 
ture, an open air temperature, ignition switch oper- 
ation, a brake pedal stroke, an accelerator pedal 
stroke, side brake operation, horn operation, belt 
loosening, a fuel remainder, light or lamp operation, 
seat belt operation, door opening, window opening, 
ABS operation, air-conditioner operation, distance 
to the preceding and succeeding vehicles and 
decrease of air pressure in tire. 

3. A device according to claim 1 or 2 wherein user's 
own data is included in the current status detected 
by said first sensor means, said user's own data 
being also considered in determining the communi- 
cation program to be executed. 

4. A device according to claim 1 , 2 or 3 which further 
comprises second sensor means for detecting 
user's response to the agent's activity outputted by 
said agent output means, and third memory means 

35 for storing users response data detected by said 
second sensor means, said user's response data 
being also considered in determining the communi- 
cation program to be executed. 

40 5. A device according to claim 4 wherein said user's 
response data is subjected to study so that said 
study data in said first memory means is also 
renewable in response to entry of the latest user's 
response data. 

45 

6. A device according to any one of claims 1 to 5 
wherein said agent output means comprises one or 
both of display means and speaker means. 

so 7. A device according to any one of claims 1 to 6 
wherein said agent output means at least includes 
speaker means, utterance pattern of the agent out- 
putted by said speaker means being selectively 
determined by said agent control means. 

55 

8. A device according to any one of claims 1 to 7 
wherein said agent output means at least includes 
display means, appearance pattern of the agent 
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outputted by said display means being selectively 
determined by said agent control' means. 

9. A device according to any one of claims 1 to 8 
which further comprises agent's character select s 
means for determining a specific character to be 
given to the agent, said agent control means deter- 
mining the agent's activity in consideration of the 
agent's character determined by said agent's char- 
acter select means. io 

10. A device according to any one of claims 1 to 9 
which further comprises fourth memory means for 
storing data indicating user's familiarity to agent 
operation of the device, said agent control means 75 
determining the agent's activity in consideration of 
the agent's familiarity stored in said fourth memory 
means. 

11. A device according to any one of claims 1 to 10 so 
wherein said agent output means allows a plurality 

of different agents to be distinguishably recognized 
one another by the user in the vehicle. 

1 2. A device according to any one of claims 1 to 1 1 25 
which further comprises fifth memory means for 
storing sub-agent program, said sub-agent pro- 
gram being selected in correspondence to a spe- 
cific communication program stored in said second 
memory means so that a sub-agent of the selected so 
sub-agent program is outputted by said agent out- 
put means together with the main agent of the exe- 
cuted communication program. 

13. A device according to claim 12 wherein said sub- 35 
agent explains the activity of the main agent. 

14. A device according to claim 12 or 13 wherein said 
sub-agent allows short-cut of operation executed by 

the main agent communication program. 40 

15. A device according to claim 12, 13, or 14 wherein 
said sub-ag^nt program becomes operative when 
data indicating user's familiarity to agent operation 

of the device is greater than a predetermined criti- 45 
cal value. ' 

16. A device according to any one of claims 1 to 15 
wherein said agent output means at least includes 
display means, a background of said display means so 
being selectively determined so that the agent 
appears on said display means together with the 
selected background. 
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FIG. 3 
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